Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [............X............X.X...X.....X...X..X....X.......X..] duration: 5.993791531s trace: https://jaeger.dev.storj.io/trace/249772df51a41f53 FAIL (2025-12-27 22:52:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 50:52.461085904 (535h21m52.332168939s) [upload=469531]: Starting upload Dec 27 22:50:52.461169103 (535h21m52.332252138s) [upload=469531]: Done starting upload Dec 27 22:50:52.461286086 (535h21m52.332369131s) [upload=469531]: Uploading segments... Dec 27 22:50:52.461467611 (535h21m52.332550656s) [upload=469531]: Got next segment. Inline: false Dec 27 22:50:52.461536713 (535h21m52.332619748s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:50:52.461543065 (535h21m52.3326261s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:50:52.710875814 (535h21m52.581958849s) [seg_pos={0 0}, upload=469531]: Begin upload segment... Dec 27 22:50:52.710930999 (535h21m52.582014044s) [seg_pos={0 0}, upload=469531]: Done begin upload segment. Dec 27 22:50:52.710999630 (535h21m52.582082675s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711050677 (535h21m52.582133722s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711498409 (535h21m52.582581454s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711773042 (535h21m52.582856077s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.712024952 (535h21m52.583107987s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713023093 (535h21m52.584106128s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713444104 (535h21m52.584527149s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.714709514 (535h21m52.585792549s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.715073497 (535h21m52.586156532s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.716633177 (535h21m52.587716212s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.717885492 (535h21m52.588968527s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.719465961 (535h21m52.590549006s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.720765626 (535h21m52.591848661s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.721174254 (535h21m52.592257289s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.722066553 (535h21m52.593149598s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.723723057 (535h21m52.594806092s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:54.424318986 (535h21m54.295402021s) [upload=469531]: Next returned nil segment Dec 27 22:50:54.424335727 (535h21m54.295418792s) [upload=469531]: Waiting for error group managing segments... Dec 27 22:50:54.704253143 (535h21m54.575336178s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.713715471 (535h21m54.584798506s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.714936987 (535h21m54.586020022s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.724954162 (535h21m54.596037197s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.733463625 (535h21m54.60454667s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.740845331 (535h21m54.611928366s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.750709694 (535h21m54.621792729s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.754624639 (535h21m54.625707674s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.848487348 (535h21m54.719570383s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.903037509 (535h21m54.774120554s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.967426356 (535h21m54.838509391s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.973908930 (535h21m54.844991975s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010728199 (535h21m54.881811245s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010747706 (535h21m54.881830741s) [seg_pos={0 0}, upload=469531]: Segment reached optimal threshold of 13 pieces. Dec 27 22:50:55.010957275 (535h21m54.88204031s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.010964570 (535h21m54.882047625s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011165852 (535h21m54.882248887s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection Dec 27 22:50:55.011171995 (535h21m54.88225504s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011223212 (535h21m54.882306257s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.011241487 (535h21m54.882324532s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011288997 (535h21m54.882372032s) [seg_pos={0 0}, upload=469531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:50:55.011361084 (535h21m54.882444129s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:50:55.011365843 (535h21m54.882448888s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:50:55.303924079 (535h21m55.175007115s) [upload=469531]: Done uploading segments... Dec 27 22:50:55.303943547 (535h21m55.175026582s) [upload=469531]: Upload finished. err: <nil> Duration: 5.993818261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.............X............X.X...X.....X...X..X....X.......X.] duration: 8.891245319s trace: https://jaeger.dev.storj.io/trace/249772df51a41f53 FAIL (2025-12-27 22:51:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 50:52.461085904 (535h21m52.332168939s) [upload=469531]: Starting upload Dec 27 22:50:52.461169103 (535h21m52.332252138s) [upload=469531]: Done starting upload Dec 27 22:50:52.461286086 (535h21m52.332369131s) [upload=469531]: Uploading segments... Dec 27 22:50:52.461467611 (535h21m52.332550656s) [upload=469531]: Got next segment. Inline: false Dec 27 22:50:52.461536713 (535h21m52.332619748s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:50:52.461543065 (535h21m52.3326261s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:50:52.710875814 (535h21m52.581958849s) [seg_pos={0 0}, upload=469531]: Begin upload segment... Dec 27 22:50:52.710930999 (535h21m52.582014044s) [seg_pos={0 0}, upload=469531]: Done begin upload segment. Dec 27 22:50:52.710999630 (535h21m52.582082675s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711050677 (535h21m52.582133722s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711498409 (535h21m52.582581454s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711773042 (535h21m52.582856077s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.712024952 (535h21m52.583107987s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713023093 (535h21m52.584106128s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713444104 (535h21m52.584527149s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.714709514 (535h21m52.585792549s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.715073497 (535h21m52.586156532s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.716633177 (535h21m52.587716212s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.717885492 (535h21m52.588968527s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.719465961 (535h21m52.590549006s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.720765626 (535h21m52.591848661s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.721174254 (535h21m52.592257289s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.722066553 (535h21m52.593149598s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.723723057 (535h21m52.594806092s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:54.424318986 (535h21m54.295402021s) [upload=469531]: Next returned nil segment Dec 27 22:50:54.424335727 (535h21m54.295418792s) [upload=469531]: Waiting for error group managing segments... Dec 27 22:50:54.704253143 (535h21m54.575336178s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.713715471 (535h21m54.584798506s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.714936987 (535h21m54.586020022s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.724954162 (535h21m54.596037197s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.733463625 (535h21m54.60454667s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.740845331 (535h21m54.611928366s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.750709694 (535h21m54.621792729s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.754624639 (535h21m54.625707674s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.848487348 (535h21m54.719570383s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.903037509 (535h21m54.774120554s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.967426356 (535h21m54.838509391s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.973908930 (535h21m54.844991975s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010728199 (535h21m54.881811245s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010747706 (535h21m54.881830741s) [seg_pos={0 0}, upload=469531]: Segment reached optimal threshold of 13 pieces. Dec 27 22:50:55.010957275 (535h21m54.88204031s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.010964570 (535h21m54.882047625s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011165852 (535h21m54.882248887s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection Dec 27 22:50:55.011171995 (535h21m54.88225504s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011223212 (535h21m54.882306257s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.011241487 (535h21m54.882324532s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011288997 (535h21m54.882372032s) [seg_pos={0 0}, upload=469531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:50:55.011361084 (535h21m54.882444129s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:50:55.011365843 (535h21m54.882448888s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:50:55.303924079 (535h21m55.175007115s) [upload=469531]: Done uploading segments... Dec 27 22:50:55.303943547 (535h21m55.175026582s) [upload=469531]: Upload finished. err: <nil> Duration: 8.891274476s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............X............X.X...X.....X...X..X....X.......X] duration: 30.002740624s trace: https://jaeger.dev.storj.io/trace/249772df51a41f53 FAIL (2025-12-27 22:51:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 50:52.461085904 (535h21m52.332168939s) [upload=469531]: Starting upload Dec 27 22:50:52.461169103 (535h21m52.332252138s) [upload=469531]: Done starting upload Dec 27 22:50:52.461286086 (535h21m52.332369131s) [upload=469531]: Uploading segments... Dec 27 22:50:52.461467611 (535h21m52.332550656s) [upload=469531]: Got next segment. Inline: false Dec 27 22:50:52.461536713 (535h21m52.332619748s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:50:52.461543065 (535h21m52.3326261s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:50:52.710875814 (535h21m52.581958849s) [seg_pos={0 0}, upload=469531]: Begin upload segment... Dec 27 22:50:52.710930999 (535h21m52.582014044s) [seg_pos={0 0}, upload=469531]: Done begin upload segment. Dec 27 22:50:52.710999630 (535h21m52.582082675s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711050677 (535h21m52.582133722s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711498409 (535h21m52.582581454s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.711773042 (535h21m52.582856077s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.712024952 (535h21m52.583107987s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713023093 (535h21m52.584106128s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.713444104 (535h21m52.584527149s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.714709514 (535h21m52.585792549s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.715073497 (535h21m52.586156532s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.716633177 (535h21m52.587716212s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.717885492 (535h21m52.588968527s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.719465961 (535h21m52.590549006s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.720765626 (535h21m52.591848661s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.721174254 (535h21m52.592257289s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.722066553 (535h21m52.593149598s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:52.723723057 (535h21m52.594806092s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Uploading piece... Dec 27 22:50:54.424318986 (535h21m54.295402021s) [upload=469531]: Next returned nil segment Dec 27 22:50:54.424335727 (535h21m54.295418792s) [upload=469531]: Waiting for error group managing segments... Dec 27 22:50:54.704253143 (535h21m54.575336178s) [noise=true , address=145.79.227.5:4223 , piece_id=YU4RZOZVT3TZ2G6ZJ34UU66WMMNCBGEWZRWJ5L3LEOKL5JPGO3MA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.713715471 (535h21m54.584798506s) [noise=true , address=145.79.227.50:4375 , piece_id=L4DQBISJCPML3YPLZHRTQEEWZVXHRD5OTIRSK7FHGUN7MLG3EH4A, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.714936987 (535h21m54.586020022s) [noise=true , address=145.79.227.9:4365 , piece_id=J2CZUJ33JOBP4ADRRCOC274VUM4YQHJMJ5ROM7N722V2TU2PQP7Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.724954162 (535h21m54.596037197s) [noise=true , address=145.79.227.4:4186 , piece_id=RR6DK7YVWGPISFOW5IH6CGLUPY3O4P5RIQDAE7HEUJEOYGJDP3CQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.733463625 (535h21m54.60454667s) [noise=true , address=145.79.227.37:4566 , piece_id=WJJNKGMY7R5BY7UCIULETMAACLO7MCQ2MMU33YFE23RF6DJVAIKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.740845331 (535h21m54.611928366s) [noise=true , address=145.79.227.51:4409 , piece_id=NGNJR2NGQUUMDGFZ6X36YSWAOEIKGQJXS5HVNGMHTAYA6W3C7CLA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.750709694 (535h21m54.621792729s) [noise=true , address=145.79.227.20:4438 , piece_id=S6QSI7NMLEILDBG7UYVXR5X64ZQ4W7T2T5ZABVC7BYABJJOS6FQQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.754624639 (535h21m54.625707674s) [noise=true , address=145.79.227.38:4979 , piece_id=HGIS6T6QHPNRKXU3SUSN5SRKBJELHXTA4UNYCRTRAZTRDIR2NPJQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.848487348 (535h21m54.719570383s) [noise=true , address=145.79.227.36:4519 , piece_id=3WPAG53ZS4CXERFZA372HFRS5BHHBG225YEXYX237A6LKXAHF3QQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.903037509 (535h21m54.774120554s) [noise=true , address=77.90.31.19:4841 , piece_id=HS4VSO4DZYCYJVDWKYOVM42XMAWR3OSESB2JK5LO7HXSCYRBQSKA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.967426356 (535h21m54.838509391s) [noise=true , address=45.91.116.5:4298 , piece_id=X4NE2X7CMNWQ6F7CO3GVI6MRKBIZYJCKIAFTCSFARLOPOTHHRSAA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:54.973908930 (535h21m54.844991975s) [noise=true , address=77.90.31.18:4808 , piece_id=EHL6RX4RYTJPBBHMSVPTAF4ZBAUUSVZGKTHIUH7PT7P3NBYK3I4Q, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010728199 (535h21m54.881811245s) [noise=true , address=45.91.116.14:4189 , piece_id=EXGI3IOO5IDKDAKGSN33UG7A4VWIYAWYBY62SRXECD2TRS4CI7WA, seg_pos={0 0}, upload=469531]: Done uploading piece. err: <nil> Dec 27 22:50:55.010747706 (535h21m54.881830741s) [seg_pos={0 0}, upload=469531]: Segment reached optimal threshold of 13 pieces. Dec 27 22:50:55.010957275 (535h21m54.88204031s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.010964570 (535h21m54.882047625s) [noise=true , address=45.91.116.12:4226 , piece_id=OVROBHIF4QTM4PPNIYUPUHJ62MQBNP4KS3CLWNAZBZNGPSGI2AFQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011165852 (535h21m54.882248887s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55554->145.79.227.17:4972: use of closed network connection Dec 27 22:50:55.011171995 (535h21m54.88225504s) [noise=true , address=145.79.227.17:4972 , piece_id=5YAYN2JAI5V34OUZVN6S42NCTBBQYOOCQKGD57SRR4VF7Y34BPZQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011223212 (535h21m54.882306257s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:50:55.011241487 (535h21m54.882324532s) [noise=true , address=45.91.116.7:4011 , piece_id=6X7XW5ZYGNLZ5DDQI75IXOFMZ6DI7Z22CO4GF7IVYECNN6445XVQ, seg_pos={0 0}, upload=469531]: Long tail cancellation due to optimal threshold Dec 27 22:50:55.011288997 (535h21m54.882372032s) [seg_pos={0 0}, upload=469531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:50:55.011361084 (535h21m54.882444129s) [upload=469531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:50:55.011365843 (535h21m54.882448888s) [upload=469531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:50:55.303924079 (535h21m55.175007115s) [upload=469531]: Done uploading segments... Dec 27 22:50:55.303943547 (535h21m55.175026582s) [upload=469531]: Upload finished. err: <nil> Duration: 30.002766273s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X............X.X...X.....X...X..X....X.......] duration: 24.896782519s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:50:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 24.896810803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X............X.X...X.....X...X..X....X......] duration: 6.331413153s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:49:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 6.331442438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X............X.X...X.....X...X..X....X.....] duration: 5.942168619s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:48:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 5.94218534s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X............X.X...X.....X...X..X....X....] duration: 6.358478357s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:47:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 6.358502523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X............X.X...X.....X...X..X....X...] duration: 25.959563503s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:47:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 25.959588891s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 208.315128ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:46:42 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 208.342079ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X............X.X...X.....X...X..X....X..] duration: 7.739269956s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:46:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 7.739296326s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X............X.X...X.....X...X..X....X.] duration: 6.099288725s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:45:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 6.099313102s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X............X.X...X.....X...X..X....X] duration: 30.001765275s trace: https://jaeger.dev.storj.io/trace/67e5b65c1836be39 FAIL (2025-12-27 22:45:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ) context deadline exceeded ======LOGS====== 16m4.620323326s) [upload=469438]: Uploading segments... Dec 27 22:45:04.749488463 (535h16m4.620571508s) [upload=469438]: Got next segment. Inline: false Dec 27 22:45:04.749529381 (535h16m4.620612436s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:45:04.749536094 (535h16m4.620619149s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:45:05.000717426 (535h16m4.871800501s) [seg_pos={0 0}, upload=469438]: Begin upload segment... Dec 27 22:45:05.000782259 (535h16m4.871865304s) [seg_pos={0 0}, upload=469438]: Done begin upload segment. Dec 27 22:45:05.000821734 (535h16m4.871904789s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.001222127 (535h16m4.872305182s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002353952 (535h16m4.873436997s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002883460 (535h16m4.873966515s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.002930680 (535h16m4.874013736s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003303590 (535h16m4.874386645s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.003723670 (535h16m4.874806715s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004172565 (535h16m4.87525561s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004194817 (535h16m4.875277872s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.004633323 (535h16m4.875716378s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.005010059 (535h16m4.876093104s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.006339952 (535h16m4.877422997s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.007234004 (535h16m4.878317049s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.011044781 (535h16m4.882127826s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.012865718 (535h16m4.883948763s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:05.014688018 (535h16m4.885771063s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Uploading piece... Dec 27 22:45:06.624796567 (535h16m6.495879612s) [upload=469438]: Next returned nil segment Dec 27 22:45:06.624814311 (535h16m6.495897356s) [upload=469438]: Waiting for error group managing segments... Dec 27 22:45:07.009031135 (535h16m6.88011418s) [noise=true , address=145.79.227.11:4731 , piece_id=6PL34PORDMFYGTF7KU4ZAHRLO7JU66BFD55FEH23EYKUQRHWTERA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.012820972 (535h16m6.883904017s) [noise=true , address=145.79.227.24:4582 , piece_id=ZJQ5GMGRKHLAUV3DZ4LYIUDP7CDJ4C4TTYIB5SK5LGECHEW2ZVSQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.019694149 (535h16m6.890777184s) [noise=true , address=145.79.227.39:4013 , piece_id=VX4DUSCEWUKC5HTCUVUNQEKOSPR4RJVTPOU7YBSWBPQRTGAZ6RNA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.020947285 (535h16m6.89203033s) [noise=true , address=145.79.227.9:4365 , piece_id=K2IZ3KP7NM6ZU4BZ6RPEO6LVCTYIM33APM2UWJHPNXGMIWVJ6CXA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.026693517 (535h16m6.897776552s) [noise=true , address=145.79.227.42:4122 , piece_id=ALADJYXJQ4TQTCFKQIN3CHH3GTU72ZB7G3WXX74LIPZN4TD3TD6A, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.027854889 (535h16m6.898937924s) [noise=true , address=145.79.227.5:4223 , piece_id=WNS3IGWVQAGLWPGEJR7474PEUBVKUWG7ISF3P73TXYE3MNCLECLA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.028395879 (535h16m6.899478924s) [noise=true , address=145.79.227.38:4977 , piece_id=ZM7RPMQIVMGTXLO55VGICXWNIEVMUEOZGWSY7HHF3OYUPUSJWEKA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.032625914 (535h16m6.903708949s) [noise=true , address=145.79.227.45:4231 , piece_id=IVFX654533G6O4WXZFAG4ONLB5LIOIVEC6CWE4S3QELFDO5KHDEQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.048007063 (535h16m6.919090098s) [noise=true , address=145.79.227.47:4266 , piece_id=DCQJDVWX7NEWG5R6JHRIUQBEMGZJLNNSBMPKMQPT3L6VOJEWK3EQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.167310827 (535h16m7.038393872s) [noise=true , address=77.90.31.14:4661 , piece_id=ZSUJHALWAQOHVMTXFW54BB6RF7Z3TVGBUYMAKZIUP4KHVZE2YSTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.221778913 (535h16m7.092861948s) [noise=true , address=45.91.116.5:4297 , piece_id=6ZI3SEX5K2F32GL3G6TZ5KZPMWOCLYQGA63JCSIUU334WFQQ3TOQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.244653280 (535h16m7.115736325s) [noise=true , address=45.91.116.11:4117 , piece_id=P2HLL4ATCVG2QIOKBJVGO2PFJQCLYA5GLURFGPHPEN4JYRJ5S5LQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250513118 (535h16m7.121596153s) [noise=true , address=45.91.116.8:4047 , piece_id=OIQP4IXTAED2DLC2X6R6TMBK546W7TXXVFVGRCV3WGDCH65T3FQA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: <nil> Dec 27 22:45:07.250536132 (535h16m7.121619177s) [seg_pos={0 0}, upload=469438]: Segment reached optimal threshold of 13 pieces. Dec 27 22:45:07.250729120 (535h16m7.121812165s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54026->145.79.227.37:4567: use of closed network connection Dec 27 22:45:07.250735572 (535h16m7.121818617s) [noise=true , address=145.79.227.37:4567 , piece_id=QHZOH2IPBXQWSBJHBJ4EBJCW5JHHQRURJM4GODY3NP2AOUFWWR5Q, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.250951833 (535h16m7.122034879s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58332->77.90.31.10:4518: use of closed network connection Dec 27 22:45:07.250961772 (535h16m7.122044807s) [noise=true , address=77.90.31.10:4518 , piece_id=SPJXKTKKCAWKWVJHK34YDRQL7BC4I4GRY4HZYUBL7WZPAT5GYGTA, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251122168 (535h16m7.122205203s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 27 22:45:07.251129642 (535h16m7.122212687s) [noise=true , address=145.79.227.49:4338 , piece_id=UPBP4WKNJB4PG5REJGTAAQEVIRLUVCQHJUV43OTXKT6DGJZGUEHQ, seg_pos={0 0}, upload=469438]: Long tail cancellation due to optimal threshold Dec 27 22:45:07.251146795 (535h16m7.12222983s) [seg_pos={0 0}, upload=469438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:45:07.251218642 (535h16m7.122301677s) [upload=469438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:45:07.251223621 (535h16m7.122306666s) [upload=469438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:45:07.559197465 (535h16m7.430280501s) [upload=469438]: Done uploading segments... Dec 27 22:45:07.559217354 (535h16m7.430300389s) [upload=469438]: Upload finished. err: <nil> Duration: 30.001793469s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 153.064362ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:45:27 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 153.094148ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 147.971634ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:45:10 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 148.002893ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......................X............X.X...X.....X...X..X....] duration: 23.70389234s trace: https://jaeger.dev.storj.io/trace/7c238302dd1e71de FAIL (2025-12-27 22:44:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A) context deadline exceeded ======LOGS====== Dec 27 22:41:06.263211046 (535h12m6.134294091s) [upload=469384]: Starting upload Dec 27 22:41:06.263315615 (535h12m6.13439866s) [upload=469384]: Done starting upload Dec 27 22:41:06.263576553 (535h12m6.134659588s) [upload=469384]: Uploading segments... Dec 27 22:41:06.263675500 (535h12m6.134758555s) [upload=469384]: Got next segment. Inline: false Dec 27 22:41:06.263742808 (535h12m6.134825843s) [upload=469384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:41:06.263749452 (535h12m6.134832487s) [upload=469384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:41:06.514179133 (535h12m6.385262188s) [seg_pos={0 0}, upload=469384]: Begin upload segment... Dec 27 22:41:06.514231803 (535h12m6.385314838s) [seg_pos={0 0}, upload=469384]: Done begin upload segment. Dec 27 22:41:06.514255659 (535h12m6.385338694s) [noise=true , address=145.79.227.15:4875 , piece_id=WRY6RL7PE2JOORJ2PHDSIM7ATFD7KBPYX3KDWRXFWZVNXDBSCUKA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.514286287 (535h12m6.385369332s) [noise=true , address=45.91.116.14:4192 , piece_id=JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.514865319 (535h12m6.385948354s) [noise=true , address=45.91.116.4:4940 , piece_id=SJKSBORR77OTAYHDWG6MNMQOEYPGP6ZOBUTK5ICJZQ3UALLRSMRA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515122970 (535h12m6.386206015s) [noise=true , address=145.79.227.45:4230 , piece_id=TH726VLYG7HBMZAW3US6UODLUFFSYXZYCADR2MZWJ5EAVFMCUEOA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515236416 (535h12m6.386319461s) [noise=true , address=45.91.116.11:4117 , piece_id=MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515534584 (535h12m6.386617619s) [noise=true , address=145.79.227.11:4732 , piece_id=SNGMU2CXUQVERANJGQTZKTU35PYZI53Z3BXZT2B55PVXTQZO6FGA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515631298 (535h12m6.386714343s) [noise=true , address=145.79.227.3:4150 , piece_id=J4ZUOGGVXETFRHLNKTE6XZWRDQWMKVNUTL7UBBQUTLARU6TBF2VQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515953171 (535h12m6.387036216s) [noise=true , address=145.79.227.19:4079 , piece_id=HQSEYGCQTOBXZQGXP7XYZKCCNRRSW6ZFNFVFUPM7WX3NTBABFTFQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516037752 (535h12m6.387120797s) [noise=true , address=145.79.227.41:4086 , piece_id=36KCVTBY5J4LYLP2K6T3YOBQ6EJMTDOEZZBZBXPB5BPABCDUPC6Q, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516490744 (535h12m6.387573779s) [noise=true , address=145.79.227.4:4186 , piece_id=AV4SQCVNTMHMZSVZDCQCHFAFHPEN6T4LAJR2Z4OJS22XHSOLUFKQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516861300 (535h12m6.387944345s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517229932 (535h12m6.388312977s) [noise=true , address=145.79.227.17:4970 , piece_id=5V432SJLJSUXZGZ47ATA4NVWAAXM3DGH4EQSXEUPFOBJ3IT4NXTA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517355000 (535h12m6.388438045s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517603083 (535h12m6.388686118s) [noise=true , address=145.79.227.21:4473 , piece_id=XEE4BEIR5CQEHPL7K2EMPDOGPQ5S3KU3RP5AS73HQFKZYRPFFMXA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.518156046 (535h12m6.389239091s) [noise=true , address=77.90.31.14:4664 , piece_id=BLO2PS37YHZXTA4NDTPSBSZM2NEMH2XVRRGEE5FM3RQNJOASSAPA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.518574112 (535h12m6.389657157s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:08.233199608 (535h12m8.104282653s) [upload=469384]: Next returned nil segment Dec 27 22:41:08.233213654 (535h12m8.104296699s) [upload=469384]: Waiting for error group managing segments... Dec 27 22:41:08.517424289 (535h12m8.388507324s) [noise=true , address=145.79.227.4:4186 , piece_id=AV4SQCVNTMHMZSVZDCQCHFAFHPEN6T4LAJR2Z4OJS22XHSOLUFKQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.518463548 (535h12m8.389546603s) [noise=true , address=145.79.227.17:4970 , piece_id=5V432SJLJSUXZGZ47ATA4NVWAAXM3DGH4EQSXEUPFOBJ3IT4NXTA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.522372302 (535h12m8.393455347s) [noise=true , address=145.79.227.41:4086 , piece_id=36KCVTBY5J4LYLP2K6T3YOBQ6EJMTDOEZZBZBXPB5BPABCDUPC6Q, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.522984537 (535h12m8.394067582s) [noise=true , address=145.79.227.19:4079 , piece_id=HQSEYGCQTOBXZQGXP7XYZKCCNRRSW6ZFNFVFUPM7WX3NTBABFTFQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.532742538 (535h12m8.403825573s) [noise=true , address=145.79.227.15:4875 , piece_id=WRY6RL7PE2JOORJ2PHDSIM7ATFD7KBPYX3KDWRXFWZVNXDBSCUKA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.536956262 (535h12m8.408039307s) [noise=true , address=145.79.227.45:4230 , piece_id=TH726VLYG7HBMZAW3US6UODLUFFSYXZYCADR2MZWJ5EAVFMCUEOA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.538742934 (535h12m8.409825969s) [noise=true , address=145.79.227.3:4150 , piece_id=J4ZUOGGVXETFRHLNKTE6XZWRDQWMKVNUTL7UBBQUTLARU6TBF2VQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.549511399 (535h12m8.420594444s) [noise=true , address=145.79.227.21:4473 , piece_id=XEE4BEIR5CQEHPL7K2EMPDOGPQ5S3KU3RP5AS73HQFKZYRPFFMXA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.553212888 (535h12m8.424295933s) [noise=true , address=145.79.227.11:4732 , piece_id=SNGMU2CXUQVERANJGQTZKTU35PYZI53Z3BXZT2B55PVXTQZO6FGA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.691583047 (535h12m8.562666082s) [noise=true , address=77.90.31.14:4664 , piece_id=BLO2PS37YHZXTA4NDTPSBSZM2NEMH2XVRRGEE5FM3RQNJOASSAPA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.761123714 (535h12m8.632206749s) [noise=true , address=45.91.116.14:4192 , piece_id=JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.764636253 (535h12m8.635719288s) [noise=true , address=45.91.116.11:4117 , piece_id=MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.788412797 (535h12m8.659495832s) [noise=true , address=45.91.116.4:4940 , piece_id=SJKSBORR77OTAYHDWG6MNMQOEYPGP6ZOBUTK5ICJZQ3UALLRSMRA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.788444768 (535h12m8.659527803s) [seg_pos={0 0}, upload=469384]: Segment reached optimal threshold of 13 pieces. Dec 27 22:41:08.788636223 (535h12m8.659719258s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788642946 (535h12m8.659725981s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788824662 (535h12m8.659907707s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788830923 (535h12m8.659913958s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788931315 (535h12m8.66001437s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788951373 (535h12m8.660034418s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788988564 (535h12m8.660071609s) [seg_pos={0 0}, upload=469384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:41:08.789110045 (535h12m8.66019309s) [upload=469384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:41:08.789118952 (535h12m8.660201997s) [upload=469384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:41:09.083206054 (535h12m8.954289089s) [upload=469384]: Done uploading segments... Dec 27 22:41:09.083224619 (535h12m8.954307654s) [upload=469384]: Upload finished. err: <nil> Duration: 23.703930663s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 172.538743ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:44:12 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 172.570183ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 138.626328ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:43:21 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 138.703946ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................X............X.X...X.....X...X..X...] duration: 6.214657196s trace: https://jaeger.dev.storj.io/trace/7c238302dd1e71de FAIL (2025-12-27 22:43:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A) context deadline exceeded ======LOGS====== Dec 27 22:41:06.263211046 (535h12m6.134294091s) [upload=469384]: Starting upload Dec 27 22:41:06.263315615 (535h12m6.13439866s) [upload=469384]: Done starting upload Dec 27 22:41:06.263576553 (535h12m6.134659588s) [upload=469384]: Uploading segments... Dec 27 22:41:06.263675500 (535h12m6.134758555s) [upload=469384]: Got next segment. Inline: false Dec 27 22:41:06.263742808 (535h12m6.134825843s) [upload=469384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:41:06.263749452 (535h12m6.134832487s) [upload=469384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:41:06.514179133 (535h12m6.385262188s) [seg_pos={0 0}, upload=469384]: Begin upload segment... Dec 27 22:41:06.514231803 (535h12m6.385314838s) [seg_pos={0 0}, upload=469384]: Done begin upload segment. Dec 27 22:41:06.514255659 (535h12m6.385338694s) [noise=true , address=145.79.227.15:4875 , piece_id=WRY6RL7PE2JOORJ2PHDSIM7ATFD7KBPYX3KDWRXFWZVNXDBSCUKA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.514286287 (535h12m6.385369332s) [noise=true , address=45.91.116.14:4192 , piece_id=JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.514865319 (535h12m6.385948354s) [noise=true , address=45.91.116.4:4940 , piece_id=SJKSBORR77OTAYHDWG6MNMQOEYPGP6ZOBUTK5ICJZQ3UALLRSMRA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515122970 (535h12m6.386206015s) [noise=true , address=145.79.227.45:4230 , piece_id=TH726VLYG7HBMZAW3US6UODLUFFSYXZYCADR2MZWJ5EAVFMCUEOA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515236416 (535h12m6.386319461s) [noise=true , address=45.91.116.11:4117 , piece_id=MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515534584 (535h12m6.386617619s) [noise=true , address=145.79.227.11:4732 , piece_id=SNGMU2CXUQVERANJGQTZKTU35PYZI53Z3BXZT2B55PVXTQZO6FGA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515631298 (535h12m6.386714343s) [noise=true , address=145.79.227.3:4150 , piece_id=J4ZUOGGVXETFRHLNKTE6XZWRDQWMKVNUTL7UBBQUTLARU6TBF2VQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515953171 (535h12m6.387036216s) [noise=true , address=145.79.227.19:4079 , piece_id=HQSEYGCQTOBXZQGXP7XYZKCCNRRSW6ZFNFVFUPM7WX3NTBABFTFQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516037752 (535h12m6.387120797s) [noise=true , address=145.79.227.41:4086 , piece_id=36KCVTBY5J4LYLP2K6T3YOBQ6EJMTDOEZZBZBXPB5BPABCDUPC6Q, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516490744 (535h12m6.387573779s) [noise=true , address=145.79.227.4:4186 , piece_id=AV4SQCVNTMHMZSVZDCQCHFAFHPEN6T4LAJR2Z4OJS22XHSOLUFKQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516861300 (535h12m6.387944345s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517229932 (535h12m6.388312977s) [noise=true , address=145.79.227.17:4970 , piece_id=5V432SJLJSUXZGZ47ATA4NVWAAXM3DGH4EQSXEUPFOBJ3IT4NXTA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517355000 (535h12m6.388438045s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517603083 (535h12m6.388686118s) [noise=true , address=145.79.227.21:4473 , piece_id=XEE4BEIR5CQEHPL7K2EMPDOGPQ5S3KU3RP5AS73HQFKZYRPFFMXA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.518156046 (535h12m6.389239091s) [noise=true , address=77.90.31.14:4664 , piece_id=BLO2PS37YHZXTA4NDTPSBSZM2NEMH2XVRRGEE5FM3RQNJOASSAPA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.518574112 (535h12m6.389657157s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:08.233199608 (535h12m8.104282653s) [upload=469384]: Next returned nil segment Dec 27 22:41:08.233213654 (535h12m8.104296699s) [upload=469384]: Waiting for error group managing segments... Dec 27 22:41:08.517424289 (535h12m8.388507324s) [noise=true , address=145.79.227.4:4186 , piece_id=AV4SQCVNTMHMZSVZDCQCHFAFHPEN6T4LAJR2Z4OJS22XHSOLUFKQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.518463548 (535h12m8.389546603s) [noise=true , address=145.79.227.17:4970 , piece_id=5V432SJLJSUXZGZ47ATA4NVWAAXM3DGH4EQSXEUPFOBJ3IT4NXTA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.522372302 (535h12m8.393455347s) [noise=true , address=145.79.227.41:4086 , piece_id=36KCVTBY5J4LYLP2K6T3YOBQ6EJMTDOEZZBZBXPB5BPABCDUPC6Q, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.522984537 (535h12m8.394067582s) [noise=true , address=145.79.227.19:4079 , piece_id=HQSEYGCQTOBXZQGXP7XYZKCCNRRSW6ZFNFVFUPM7WX3NTBABFTFQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.532742538 (535h12m8.403825573s) [noise=true , address=145.79.227.15:4875 , piece_id=WRY6RL7PE2JOORJ2PHDSIM7ATFD7KBPYX3KDWRXFWZVNXDBSCUKA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.536956262 (535h12m8.408039307s) [noise=true , address=145.79.227.45:4230 , piece_id=TH726VLYG7HBMZAW3US6UODLUFFSYXZYCADR2MZWJ5EAVFMCUEOA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.538742934 (535h12m8.409825969s) [noise=true , address=145.79.227.3:4150 , piece_id=J4ZUOGGVXETFRHLNKTE6XZWRDQWMKVNUTL7UBBQUTLARU6TBF2VQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.549511399 (535h12m8.420594444s) [noise=true , address=145.79.227.21:4473 , piece_id=XEE4BEIR5CQEHPL7K2EMPDOGPQ5S3KU3RP5AS73HQFKZYRPFFMXA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.553212888 (535h12m8.424295933s) [noise=true , address=145.79.227.11:4732 , piece_id=SNGMU2CXUQVERANJGQTZKTU35PYZI53Z3BXZT2B55PVXTQZO6FGA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.691583047 (535h12m8.562666082s) [noise=true , address=77.90.31.14:4664 , piece_id=BLO2PS37YHZXTA4NDTPSBSZM2NEMH2XVRRGEE5FM3RQNJOASSAPA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.761123714 (535h12m8.632206749s) [noise=true , address=45.91.116.14:4192 , piece_id=JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.764636253 (535h12m8.635719288s) [noise=true , address=45.91.116.11:4117 , piece_id=MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.788412797 (535h12m8.659495832s) [noise=true , address=45.91.116.4:4940 , piece_id=SJKSBORR77OTAYHDWG6MNMQOEYPGP6ZOBUTK5ICJZQ3UALLRSMRA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.788444768 (535h12m8.659527803s) [seg_pos={0 0}, upload=469384]: Segment reached optimal threshold of 13 pieces. Dec 27 22:41:08.788636223 (535h12m8.659719258s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788642946 (535h12m8.659725981s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788824662 (535h12m8.659907707s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788830923 (535h12m8.659913958s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788931315 (535h12m8.66001437s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788951373 (535h12m8.660034418s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788988564 (535h12m8.660071609s) [seg_pos={0 0}, upload=469384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:41:08.789110045 (535h12m8.66019309s) [upload=469384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:41:08.789118952 (535h12m8.660201997s) [upload=469384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:41:09.083206054 (535h12m8.954289089s) [upload=469384]: Done uploading segments... Dec 27 22:41:09.083224619 (535h12m8.954307654s) [upload=469384]: Upload finished. err: <nil> Duration: 6.214683377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X............X.X...X.....X...X..X..] duration: 5.909353595s trace: https://jaeger.dev.storj.io/trace/7c238302dd1e71de FAIL (2025-12-27 22:42:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A) context deadline exceeded ======LOGS====== Dec 27 22:41:06.263211046 (535h12m6.134294091s) [upload=469384]: Starting upload Dec 27 22:41:06.263315615 (535h12m6.13439866s) [upload=469384]: Done starting upload Dec 27 22:41:06.263576553 (535h12m6.134659588s) [upload=469384]: Uploading segments... Dec 27 22:41:06.263675500 (535h12m6.134758555s) [upload=469384]: Got next segment. Inline: false Dec 27 22:41:06.263742808 (535h12m6.134825843s) [upload=469384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:41:06.263749452 (535h12m6.134832487s) [upload=469384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:41:06.514179133 (535h12m6.385262188s) [seg_pos={0 0}, upload=469384]: Begin upload segment... Dec 27 22:41:06.514231803 (535h12m6.385314838s) [seg_pos={0 0}, upload=469384]: Done begin upload segment. Dec 27 22:41:06.514255659 (535h12m6.385338694s) [noise=true , address=145.79.227.15:4875 , piece_id=WRY6RL7PE2JOORJ2PHDSIM7ATFD7KBPYX3KDWRXFWZVNXDBSCUKA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.514286287 (535h12m6.385369332s) [noise=true , address=45.91.116.14:4192 , piece_id=JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.514865319 (535h12m6.385948354s) [noise=true , address=45.91.116.4:4940 , piece_id=SJKSBORR77OTAYHDWG6MNMQOEYPGP6ZOBUTK5ICJZQ3UALLRSMRA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515122970 (535h12m6.386206015s) [noise=true , address=145.79.227.45:4230 , piece_id=TH726VLYG7HBMZAW3US6UODLUFFSYXZYCADR2MZWJ5EAVFMCUEOA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515236416 (535h12m6.386319461s) [noise=true , address=45.91.116.11:4117 , piece_id=MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515534584 (535h12m6.386617619s) [noise=true , address=145.79.227.11:4732 , piece_id=SNGMU2CXUQVERANJGQTZKTU35PYZI53Z3BXZT2B55PVXTQZO6FGA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515631298 (535h12m6.386714343s) [noise=true , address=145.79.227.3:4150 , piece_id=J4ZUOGGVXETFRHLNKTE6XZWRDQWMKVNUTL7UBBQUTLARU6TBF2VQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515953171 (535h12m6.387036216s) [noise=true , address=145.79.227.19:4079 , piece_id=HQSEYGCQTOBXZQGXP7XYZKCCNRRSW6ZFNFVFUPM7WX3NTBABFTFQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516037752 (535h12m6.387120797s) [noise=true , address=145.79.227.41:4086 , piece_id=36KCVTBY5J4LYLP2K6T3YOBQ6EJMTDOEZZBZBXPB5BPABCDUPC6Q, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516490744 (535h12m6.387573779s) [noise=true , address=145.79.227.4:4186 , piece_id=AV4SQCVNTMHMZSVZDCQCHFAFHPEN6T4LAJR2Z4OJS22XHSOLUFKQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516861300 (535h12m6.387944345s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517229932 (535h12m6.388312977s) [noise=true , address=145.79.227.17:4970 , piece_id=5V432SJLJSUXZGZ47ATA4NVWAAXM3DGH4EQSXEUPFOBJ3IT4NXTA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517355000 (535h12m6.388438045s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517603083 (535h12m6.388686118s) [noise=true , address=145.79.227.21:4473 , piece_id=XEE4BEIR5CQEHPL7K2EMPDOGPQ5S3KU3RP5AS73HQFKZYRPFFMXA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.518156046 (535h12m6.389239091s) [noise=true , address=77.90.31.14:4664 , piece_id=BLO2PS37YHZXTA4NDTPSBSZM2NEMH2XVRRGEE5FM3RQNJOASSAPA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.518574112 (535h12m6.389657157s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:08.233199608 (535h12m8.104282653s) [upload=469384]: Next returned nil segment Dec 27 22:41:08.233213654 (535h12m8.104296699s) [upload=469384]: Waiting for error group managing segments... Dec 27 22:41:08.517424289 (535h12m8.388507324s) [noise=true , address=145.79.227.4:4186 , piece_id=AV4SQCVNTMHMZSVZDCQCHFAFHPEN6T4LAJR2Z4OJS22XHSOLUFKQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.518463548 (535h12m8.389546603s) [noise=true , address=145.79.227.17:4970 , piece_id=5V432SJLJSUXZGZ47ATA4NVWAAXM3DGH4EQSXEUPFOBJ3IT4NXTA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.522372302 (535h12m8.393455347s) [noise=true , address=145.79.227.41:4086 , piece_id=36KCVTBY5J4LYLP2K6T3YOBQ6EJMTDOEZZBZBXPB5BPABCDUPC6Q, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.522984537 (535h12m8.394067582s) [noise=true , address=145.79.227.19:4079 , piece_id=HQSEYGCQTOBXZQGXP7XYZKCCNRRSW6ZFNFVFUPM7WX3NTBABFTFQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.532742538 (535h12m8.403825573s) [noise=true , address=145.79.227.15:4875 , piece_id=WRY6RL7PE2JOORJ2PHDSIM7ATFD7KBPYX3KDWRXFWZVNXDBSCUKA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.536956262 (535h12m8.408039307s) [noise=true , address=145.79.227.45:4230 , piece_id=TH726VLYG7HBMZAW3US6UODLUFFSYXZYCADR2MZWJ5EAVFMCUEOA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.538742934 (535h12m8.409825969s) [noise=true , address=145.79.227.3:4150 , piece_id=J4ZUOGGVXETFRHLNKTE6XZWRDQWMKVNUTL7UBBQUTLARU6TBF2VQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.549511399 (535h12m8.420594444s) [noise=true , address=145.79.227.21:4473 , piece_id=XEE4BEIR5CQEHPL7K2EMPDOGPQ5S3KU3RP5AS73HQFKZYRPFFMXA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.553212888 (535h12m8.424295933s) [noise=true , address=145.79.227.11:4732 , piece_id=SNGMU2CXUQVERANJGQTZKTU35PYZI53Z3BXZT2B55PVXTQZO6FGA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.691583047 (535h12m8.562666082s) [noise=true , address=77.90.31.14:4664 , piece_id=BLO2PS37YHZXTA4NDTPSBSZM2NEMH2XVRRGEE5FM3RQNJOASSAPA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.761123714 (535h12m8.632206749s) [noise=true , address=45.91.116.14:4192 , piece_id=JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.764636253 (535h12m8.635719288s) [noise=true , address=45.91.116.11:4117 , piece_id=MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.788412797 (535h12m8.659495832s) [noise=true , address=45.91.116.4:4940 , piece_id=SJKSBORR77OTAYHDWG6MNMQOEYPGP6ZOBUTK5ICJZQ3UALLRSMRA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.788444768 (535h12m8.659527803s) [seg_pos={0 0}, upload=469384]: Segment reached optimal threshold of 13 pieces. Dec 27 22:41:08.788636223 (535h12m8.659719258s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788642946 (535h12m8.659725981s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788824662 (535h12m8.659907707s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788830923 (535h12m8.659913958s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788931315 (535h12m8.66001437s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788951373 (535h12m8.660034418s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788988564 (535h12m8.660071609s) [seg_pos={0 0}, upload=469384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:41:08.789110045 (535h12m8.66019309s) [upload=469384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:41:08.789118952 (535h12m8.660201997s) [upload=469384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:41:09.083206054 (535h12m8.954289089s) [upload=469384]: Done uploading segments... Dec 27 22:41:09.083224619 (535h12m8.954307654s) [upload=469384]: Upload finished. err: <nil> Duration: 5.909380916s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 164.39697ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:42:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 164.429052ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................X............X.X...X.....X...X..X.] duration: 5.873110544s trace: https://jaeger.dev.storj.io/trace/7c238302dd1e71de FAIL (2025-12-27 22:41:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A) context deadline exceeded ======LOGS====== Dec 27 22:41:06.263211046 (535h12m6.134294091s) [upload=469384]: Starting upload Dec 27 22:41:06.263315615 (535h12m6.13439866s) [upload=469384]: Done starting upload Dec 27 22:41:06.263576553 (535h12m6.134659588s) [upload=469384]: Uploading segments... Dec 27 22:41:06.263675500 (535h12m6.134758555s) [upload=469384]: Got next segment. Inline: false Dec 27 22:41:06.263742808 (535h12m6.134825843s) [upload=469384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:41:06.263749452 (535h12m6.134832487s) [upload=469384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:41:06.514179133 (535h12m6.385262188s) [seg_pos={0 0}, upload=469384]: Begin upload segment... Dec 27 22:41:06.514231803 (535h12m6.385314838s) [seg_pos={0 0}, upload=469384]: Done begin upload segment. Dec 27 22:41:06.514255659 (535h12m6.385338694s) [noise=true , address=145.79.227.15:4875 , piece_id=WRY6RL7PE2JOORJ2PHDSIM7ATFD7KBPYX3KDWRXFWZVNXDBSCUKA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.514286287 (535h12m6.385369332s) [noise=true , address=45.91.116.14:4192 , piece_id=JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.514865319 (535h12m6.385948354s) [noise=true , address=45.91.116.4:4940 , piece_id=SJKSBORR77OTAYHDWG6MNMQOEYPGP6ZOBUTK5ICJZQ3UALLRSMRA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515122970 (535h12m6.386206015s) [noise=true , address=145.79.227.45:4230 , piece_id=TH726VLYG7HBMZAW3US6UODLUFFSYXZYCADR2MZWJ5EAVFMCUEOA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515236416 (535h12m6.386319461s) [noise=true , address=45.91.116.11:4117 , piece_id=MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515534584 (535h12m6.386617619s) [noise=true , address=145.79.227.11:4732 , piece_id=SNGMU2CXUQVERANJGQTZKTU35PYZI53Z3BXZT2B55PVXTQZO6FGA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515631298 (535h12m6.386714343s) [noise=true , address=145.79.227.3:4150 , piece_id=J4ZUOGGVXETFRHLNKTE6XZWRDQWMKVNUTL7UBBQUTLARU6TBF2VQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515953171 (535h12m6.387036216s) [noise=true , address=145.79.227.19:4079 , piece_id=HQSEYGCQTOBXZQGXP7XYZKCCNRRSW6ZFNFVFUPM7WX3NTBABFTFQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516037752 (535h12m6.387120797s) [noise=true , address=145.79.227.41:4086 , piece_id=36KCVTBY5J4LYLP2K6T3YOBQ6EJMTDOEZZBZBXPB5BPABCDUPC6Q, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516490744 (535h12m6.387573779s) [noise=true , address=145.79.227.4:4186 , piece_id=AV4SQCVNTMHMZSVZDCQCHFAFHPEN6T4LAJR2Z4OJS22XHSOLUFKQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516861300 (535h12m6.387944345s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517229932 (535h12m6.388312977s) [noise=true , address=145.79.227.17:4970 , piece_id=5V432SJLJSUXZGZ47ATA4NVWAAXM3DGH4EQSXEUPFOBJ3IT4NXTA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517355000 (535h12m6.388438045s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517603083 (535h12m6.388686118s) [noise=true , address=145.79.227.21:4473 , piece_id=XEE4BEIR5CQEHPL7K2EMPDOGPQ5S3KU3RP5AS73HQFKZYRPFFMXA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.518156046 (535h12m6.389239091s) [noise=true , address=77.90.31.14:4664 , piece_id=BLO2PS37YHZXTA4NDTPSBSZM2NEMH2XVRRGEE5FM3RQNJOASSAPA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.518574112 (535h12m6.389657157s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:08.233199608 (535h12m8.104282653s) [upload=469384]: Next returned nil segment Dec 27 22:41:08.233213654 (535h12m8.104296699s) [upload=469384]: Waiting for error group managing segments... Dec 27 22:41:08.517424289 (535h12m8.388507324s) [noise=true , address=145.79.227.4:4186 , piece_id=AV4SQCVNTMHMZSVZDCQCHFAFHPEN6T4LAJR2Z4OJS22XHSOLUFKQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.518463548 (535h12m8.389546603s) [noise=true , address=145.79.227.17:4970 , piece_id=5V432SJLJSUXZGZ47ATA4NVWAAXM3DGH4EQSXEUPFOBJ3IT4NXTA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.522372302 (535h12m8.393455347s) [noise=true , address=145.79.227.41:4086 , piece_id=36KCVTBY5J4LYLP2K6T3YOBQ6EJMTDOEZZBZBXPB5BPABCDUPC6Q, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.522984537 (535h12m8.394067582s) [noise=true , address=145.79.227.19:4079 , piece_id=HQSEYGCQTOBXZQGXP7XYZKCCNRRSW6ZFNFVFUPM7WX3NTBABFTFQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.532742538 (535h12m8.403825573s) [noise=true , address=145.79.227.15:4875 , piece_id=WRY6RL7PE2JOORJ2PHDSIM7ATFD7KBPYX3KDWRXFWZVNXDBSCUKA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.536956262 (535h12m8.408039307s) [noise=true , address=145.79.227.45:4230 , piece_id=TH726VLYG7HBMZAW3US6UODLUFFSYXZYCADR2MZWJ5EAVFMCUEOA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.538742934 (535h12m8.409825969s) [noise=true , address=145.79.227.3:4150 , piece_id=J4ZUOGGVXETFRHLNKTE6XZWRDQWMKVNUTL7UBBQUTLARU6TBF2VQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.549511399 (535h12m8.420594444s) [noise=true , address=145.79.227.21:4473 , piece_id=XEE4BEIR5CQEHPL7K2EMPDOGPQ5S3KU3RP5AS73HQFKZYRPFFMXA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.553212888 (535h12m8.424295933s) [noise=true , address=145.79.227.11:4732 , piece_id=SNGMU2CXUQVERANJGQTZKTU35PYZI53Z3BXZT2B55PVXTQZO6FGA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.691583047 (535h12m8.562666082s) [noise=true , address=77.90.31.14:4664 , piece_id=BLO2PS37YHZXTA4NDTPSBSZM2NEMH2XVRRGEE5FM3RQNJOASSAPA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.761123714 (535h12m8.632206749s) [noise=true , address=45.91.116.14:4192 , piece_id=JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.764636253 (535h12m8.635719288s) [noise=true , address=45.91.116.11:4117 , piece_id=MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.788412797 (535h12m8.659495832s) [noise=true , address=45.91.116.4:4940 , piece_id=SJKSBORR77OTAYHDWG6MNMQOEYPGP6ZOBUTK5ICJZQ3UALLRSMRA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.788444768 (535h12m8.659527803s) [seg_pos={0 0}, upload=469384]: Segment reached optimal threshold of 13 pieces. Dec 27 22:41:08.788636223 (535h12m8.659719258s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788642946 (535h12m8.659725981s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788824662 (535h12m8.659907707s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788830923 (535h12m8.659913958s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788931315 (535h12m8.66001437s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788951373 (535h12m8.660034418s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788988564 (535h12m8.660071609s) [seg_pos={0 0}, upload=469384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:41:08.789110045 (535h12m8.66019309s) [upload=469384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:41:08.789118952 (535h12m8.660201997s) [upload=469384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:41:09.083206054 (535h12m8.954289089s) [upload=469384]: Done uploading segments... Dec 27 22:41:09.083224619 (535h12m8.954307654s) [upload=469384]: Upload finished. err: <nil> Duration: 5.873190267s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 140.221686ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:41:36 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 140.248186ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X............X.X...X.....X...X..X] duration: 30.003389389s trace: https://jaeger.dev.storj.io/trace/7c238302dd1e71de FAIL (2025-12-27 22:41:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A) context deadline exceeded ======LOGS====== Dec 27 22:41:06.263211046 (535h12m6.134294091s) [upload=469384]: Starting upload Dec 27 22:41:06.263315615 (535h12m6.13439866s) [upload=469384]: Done starting upload Dec 27 22:41:06.263576553 (535h12m6.134659588s) [upload=469384]: Uploading segments... Dec 27 22:41:06.263675500 (535h12m6.134758555s) [upload=469384]: Got next segment. Inline: false Dec 27 22:41:06.263742808 (535h12m6.134825843s) [upload=469384]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:41:06.263749452 (535h12m6.134832487s) [upload=469384]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:41:06.514179133 (535h12m6.385262188s) [seg_pos={0 0}, upload=469384]: Begin upload segment... Dec 27 22:41:06.514231803 (535h12m6.385314838s) [seg_pos={0 0}, upload=469384]: Done begin upload segment. Dec 27 22:41:06.514255659 (535h12m6.385338694s) [noise=true , address=145.79.227.15:4875 , piece_id=WRY6RL7PE2JOORJ2PHDSIM7ATFD7KBPYX3KDWRXFWZVNXDBSCUKA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.514286287 (535h12m6.385369332s) [noise=true , address=45.91.116.14:4192 , piece_id=JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.514865319 (535h12m6.385948354s) [noise=true , address=45.91.116.4:4940 , piece_id=SJKSBORR77OTAYHDWG6MNMQOEYPGP6ZOBUTK5ICJZQ3UALLRSMRA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515122970 (535h12m6.386206015s) [noise=true , address=145.79.227.45:4230 , piece_id=TH726VLYG7HBMZAW3US6UODLUFFSYXZYCADR2MZWJ5EAVFMCUEOA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515236416 (535h12m6.386319461s) [noise=true , address=45.91.116.11:4117 , piece_id=MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515534584 (535h12m6.386617619s) [noise=true , address=145.79.227.11:4732 , piece_id=SNGMU2CXUQVERANJGQTZKTU35PYZI53Z3BXZT2B55PVXTQZO6FGA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515631298 (535h12m6.386714343s) [noise=true , address=145.79.227.3:4150 , piece_id=J4ZUOGGVXETFRHLNKTE6XZWRDQWMKVNUTL7UBBQUTLARU6TBF2VQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.515953171 (535h12m6.387036216s) [noise=true , address=145.79.227.19:4079 , piece_id=HQSEYGCQTOBXZQGXP7XYZKCCNRRSW6ZFNFVFUPM7WX3NTBABFTFQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516037752 (535h12m6.387120797s) [noise=true , address=145.79.227.41:4086 , piece_id=36KCVTBY5J4LYLP2K6T3YOBQ6EJMTDOEZZBZBXPB5BPABCDUPC6Q, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516490744 (535h12m6.387573779s) [noise=true , address=145.79.227.4:4186 , piece_id=AV4SQCVNTMHMZSVZDCQCHFAFHPEN6T4LAJR2Z4OJS22XHSOLUFKQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.516861300 (535h12m6.387944345s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517229932 (535h12m6.388312977s) [noise=true , address=145.79.227.17:4970 , piece_id=5V432SJLJSUXZGZ47ATA4NVWAAXM3DGH4EQSXEUPFOBJ3IT4NXTA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517355000 (535h12m6.388438045s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.517603083 (535h12m6.388686118s) [noise=true , address=145.79.227.21:4473 , piece_id=XEE4BEIR5CQEHPL7K2EMPDOGPQ5S3KU3RP5AS73HQFKZYRPFFMXA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.518156046 (535h12m6.389239091s) [noise=true , address=77.90.31.14:4664 , piece_id=BLO2PS37YHZXTA4NDTPSBSZM2NEMH2XVRRGEE5FM3RQNJOASSAPA, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:06.518574112 (535h12m6.389657157s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Uploading piece... Dec 27 22:41:08.233199608 (535h12m8.104282653s) [upload=469384]: Next returned nil segment Dec 27 22:41:08.233213654 (535h12m8.104296699s) [upload=469384]: Waiting for error group managing segments... Dec 27 22:41:08.517424289 (535h12m8.388507324s) [noise=true , address=145.79.227.4:4186 , piece_id=AV4SQCVNTMHMZSVZDCQCHFAFHPEN6T4LAJR2Z4OJS22XHSOLUFKQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.518463548 (535h12m8.389546603s) [noise=true , address=145.79.227.17:4970 , piece_id=5V432SJLJSUXZGZ47ATA4NVWAAXM3DGH4EQSXEUPFOBJ3IT4NXTA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.522372302 (535h12m8.393455347s) [noise=true , address=145.79.227.41:4086 , piece_id=36KCVTBY5J4LYLP2K6T3YOBQ6EJMTDOEZZBZBXPB5BPABCDUPC6Q, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.522984537 (535h12m8.394067582s) [noise=true , address=145.79.227.19:4079 , piece_id=HQSEYGCQTOBXZQGXP7XYZKCCNRRSW6ZFNFVFUPM7WX3NTBABFTFQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.532742538 (535h12m8.403825573s) [noise=true , address=145.79.227.15:4875 , piece_id=WRY6RL7PE2JOORJ2PHDSIM7ATFD7KBPYX3KDWRXFWZVNXDBSCUKA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.536956262 (535h12m8.408039307s) [noise=true , address=145.79.227.45:4230 , piece_id=TH726VLYG7HBMZAW3US6UODLUFFSYXZYCADR2MZWJ5EAVFMCUEOA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.538742934 (535h12m8.409825969s) [noise=true , address=145.79.227.3:4150 , piece_id=J4ZUOGGVXETFRHLNKTE6XZWRDQWMKVNUTL7UBBQUTLARU6TBF2VQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.549511399 (535h12m8.420594444s) [noise=true , address=145.79.227.21:4473 , piece_id=XEE4BEIR5CQEHPL7K2EMPDOGPQ5S3KU3RP5AS73HQFKZYRPFFMXA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.553212888 (535h12m8.424295933s) [noise=true , address=145.79.227.11:4732 , piece_id=SNGMU2CXUQVERANJGQTZKTU35PYZI53Z3BXZT2B55PVXTQZO6FGA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.691583047 (535h12m8.562666082s) [noise=true , address=77.90.31.14:4664 , piece_id=BLO2PS37YHZXTA4NDTPSBSZM2NEMH2XVRRGEE5FM3RQNJOASSAPA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.761123714 (535h12m8.632206749s) [noise=true , address=45.91.116.14:4192 , piece_id=JHDBDEMPRONWYZK6YJBMFBE4PCTV7RQ3E2CXXBVDXA4CB4ZNP2PQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.764636253 (535h12m8.635719288s) [noise=true , address=45.91.116.11:4117 , piece_id=MKKHWEJMJ6A3ULWZABT7RO7CPZWBEJ5N6OFC2YPP4JDTOB4Y524A, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.788412797 (535h12m8.659495832s) [noise=true , address=45.91.116.4:4940 , piece_id=SJKSBORR77OTAYHDWG6MNMQOEYPGP6ZOBUTK5ICJZQ3UALLRSMRA, seg_pos={0 0}, upload=469384]: Done uploading piece. err: <nil> Dec 27 22:41:08.788444768 (535h12m8.659527803s) [seg_pos={0 0}, upload=469384]: Segment reached optimal threshold of 13 pieces. Dec 27 22:41:08.788636223 (535h12m8.659719258s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788642946 (535h12m8.659725981s) [noise=true , address=45.91.116.6:4976 , piece_id=KPPLIIOB5FTIFO67BRXW3TWEIMC2ABXUCQY4FYIUCMPG7M6JQCXQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788824662 (535h12m8.659907707s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788830923 (535h12m8.659913958s) [noise=true , address=145.79.227.30:4762 , piece_id=I4FPDA2RQ6BPWWIKO5KCKHULNEJVHZ5RXCWNZZSOHQANJKE3QOWQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788931315 (535h12m8.66001437s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 27 22:41:08.788951373 (535h12m8.660034418s) [noise=true , address=45.91.116.7:4011 , piece_id=DGLJIZFNHFVAPJEMRJIN263KWODEV55OPJJ7J7RO4F2FTGHD3ZNQ, seg_pos={0 0}, upload=469384]: Long tail cancellation due to optimal threshold Dec 27 22:41:08.788988564 (535h12m8.660071609s) [seg_pos={0 0}, upload=469384]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:41:08.789110045 (535h12m8.66019309s) [upload=469384]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:41:08.789118952 (535h12m8.660201997s) [upload=469384]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:41:09.083206054 (535h12m8.954289089s) [upload=469384]: Done uploading segments... Dec 27 22:41:09.083224619 (535h12m8.954307654s) [upload=469384]: Upload finished. err: <nil> Duration: 30.003415328s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 148.454281ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:41:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 148.487014ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 165.220097ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:40:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 165.248561ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................X............X.X...X.....X...X..] duration: 20.741103913s trace: https://jaeger.dev.storj.io/trace/1971cebe04e469c5 FAIL (2025-12-27 22:40:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 22:38:55.674939949 (535h9m55.546022984s) [upload=469351]: Starting upload Dec 27 22:38:55.675021665 (535h9m55.54610473s) [upload=469351]: Done starting upload Dec 27 22:38:55.675133347 (535h9m55.546216392s) [upload=469351]: Uploading segments... Dec 27 22:38:55.675243046 (535h9m55.546326081s) [upload=469351]: Got next segment. Inline: false Dec 27 22:38:55.675277822 (535h9m55.546360857s) [upload=469351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:38:55.675282120 (535h9m55.546365155s) [upload=469351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:38:55.915329712 (535h9m55.786412757s) [seg_pos={0 0}, upload=469351]: Begin upload segment... Dec 27 22:38:55.915375059 (535h9m55.786458104s) [seg_pos={0 0}, upload=469351]: Done begin upload segment. Dec 27 22:38:55.915400928 (535h9m55.786483973s) [noise=true , address=45.91.116.11:4118 , piece_id=RJCXC4FLVAIHWSZGMCA324KIDU2QND2PZ4HUZUTUDSKSFUAS3PKQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.915486932 (535h9m55.786569967s) [noise=true , address=77.90.31.14:4663 , piece_id=JCSKZEMHZPHENKILKR4BXB3IQ3DZV2GVROA3PYF26GMVQWC65ZRA, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.915954061 (535h9m55.787037106s) [noise=true , address=145.79.227.11:4729 , piece_id=YA3KINGOLGQHSRNTZBVVE42FUG4STIXNZZTQHWT7UXD6WP2ZTCDQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.916238022 (535h9m55.787321067s) [noise=true , address=45.91.116.5:4300 , piece_id=B3LAZ3VLK26ACE5O4OOF3FKNJKTL4SO5NZL6GTEHDX2RA22BCDXQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.916589411 (535h9m55.787672456s) [noise=true , address=145.79.227.14:4838 , piece_id=NE4MDMHA2T7T2QHKSQ5D7EY773EBQNRKE63PJG5DD2VLANQNQ6JA, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.916883972 (535h9m55.787967017s) [noise=true , address=45.91.116.13:4153 , piece_id=O3ARD6XVJOW4QWZZN3O7D74H6P2ZEFEKEGWWT6XAPP6UEYAY7UYQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.917175526 (535h9m55.788258561s) [noise=true , address=145.79.227.40:4051 , piece_id=7BJV5NHIVUPFKB5BLGSJXRDKWOKN43VLPTI2UPC7LRGSQ2U7VGQQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.917465209 (535h9m55.788548244s) [noise=true , address=145.79.227.46:4603 , piece_id=6XA47UBATOH5YKTUBSSYK4IML3YJQTTL3LTFP5LKYTJ3A273JX5A, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.917793784 (535h9m55.788876819s) [noise=true , address=145.79.227.13:4803 , piece_id=AFNTM5MXCJ6LJ6LPV7AMISNDYIL3AIRG7G3YJ4GYD3QJNRAKQVGQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.919206915 (535h9m55.79028995s) [noise=true , address=145.79.227.38:4978 , piece_id=MXONAWQCQ7XHL26GUTTWRB5NIO4VDTNUMTOBJUNIM77O3EMLRWKQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.920328821 (535h9m55.791411856s) [noise=true , address=145.79.227.18:4007 , piece_id=FNZAT4SEE5CQ2RAJWX2CTEDEHM3JPHNUJGEGSCPZ77YQZDLJIWGQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.921386004 (535h9m55.792469039s) [noise=true , address=145.79.227.2:4113 , piece_id=HNSWCV5G3DPBFVEIJYST4KGZPOSXEYYDMQJMUH4BEPZVUAR2ENRQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.922920596 (535h9m55.794003641s) [noise=true , address=145.79.227.4:4186 , piece_id=2A4YKAQMFNIRXSRG6MB762I57D2LSITKO4ANABPOLX5ZQGHUQNNQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.924434559 (535h9m55.795517604s) [noise=true , address=145.79.227.32:4836 , piece_id=C32FXTZYV6PKKHEEJTISNBXKY4YPZVWEZRXAMWHM4F75TMAWGEWA, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.925750875 (535h9m55.79683391s) [noise=true , address=145.79.227.21:4473 , piece_id=BJPDSF7BDNBUSBQITKBKW35OJB6YXLXCOM5JCWJXFAKYSAEQBLVQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.926548454 (535h9m55.797631499s) [noise=true , address=77.90.31.11:4555 , piece_id=MXVALXCILZXUQ42LIUYGVQMJMBQOPQXD3KRFWP76DA6QUSLDJIQQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:57.534201006 (535h9m57.405284041s) [upload=469351]: Next returned nil segment Dec 27 22:38:57.534220764 (535h9m57.405303799s) [upload=469351]: Waiting for error group managing segments... Dec 27 22:38:57.927837958 (535h9m57.798920993s) [noise=true , address=145.79.227.38:4978 , piece_id=MXONAWQCQ7XHL26GUTTWRB5NIO4VDTNUMTOBJUNIM77O3EMLRWKQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.928833213 (535h9m57.799916258s) [noise=true , address=145.79.227.14:4838 , piece_id=NE4MDMHA2T7T2QHKSQ5D7EY773EBQNRKE63PJG5DD2VLANQNQ6JA, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.933927804 (535h9m57.805010839s) [noise=true , address=145.79.227.21:4473 , piece_id=BJPDSF7BDNBUSBQITKBKW35OJB6YXLXCOM5JCWJXFAKYSAEQBLVQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.944054576 (535h9m57.815137611s) [noise=true , address=145.79.227.4:4186 , piece_id=2A4YKAQMFNIRXSRG6MB762I57D2LSITKO4ANABPOLX5ZQGHUQNNQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.945714578 (535h9m57.816797613s) [noise=true , address=145.79.227.18:4007 , piece_id=FNZAT4SEE5CQ2RAJWX2CTEDEHM3JPHNUJGEGSCPZ77YQZDLJIWGQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.946069714 (535h9m57.817152759s) [noise=true , address=145.79.227.11:4729 , piece_id=YA3KINGOLGQHSRNTZBVVE42FUG4STIXNZZTQHWT7UXD6WP2ZTCDQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.951466740 (535h9m57.822549776s) [noise=true , address=145.79.227.46:4603 , piece_id=6XA47UBATOH5YKTUBSSYK4IML3YJQTTL3LTFP5LKYTJ3A273JX5A, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.951553035 (535h9m57.82263608s) [noise=true , address=145.79.227.32:4836 , piece_id=C32FXTZYV6PKKHEEJTISNBXKY4YPZVWEZRXAMWHM4F75TMAWGEWA, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.951902110 (535h9m57.822985145s) [noise=true , address=145.79.227.13:4803 , piece_id=AFNTM5MXCJ6LJ6LPV7AMISNDYIL3AIRG7G3YJ4GYD3QJNRAKQVGQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.957315117 (535h9m57.828398152s) [noise=true , address=145.79.227.2:4113 , piece_id=HNSWCV5G3DPBFVEIJYST4KGZPOSXEYYDMQJMUH4BEPZVUAR2ENRQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.121309660 (535h9m57.992392695s) [noise=true , address=77.90.31.14:4663 , piece_id=JCSKZEMHZPHENKILKR4BXB3IQ3DZV2GVROA3PYF26GMVQWC65ZRA, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.137653071 (535h9m58.008736106s) [noise=true , address=45.91.116.13:4153 , piece_id=O3ARD6XVJOW4QWZZN3O7D74H6P2ZEFEKEGWWT6XAPP6UEYAY7UYQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.165074426 (535h9m58.036157461s) [noise=true , address=45.91.116.5:4300 , piece_id=B3LAZ3VLK26ACE5O4OOF3FKNJKTL4SO5NZL6GTEHDX2RA22BCDXQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.165093021 (535h9m58.036176056s) [seg_pos={0 0}, upload=469351]: Segment reached optimal threshold of 13 pieces. Dec 27 22:38:58.165266581 (535h9m58.036349617s) [noise=true , address=145.79.227.40:4051 , piece_id=7BJV5NHIVUPFKB5BLGSJXRDKWOKN43VLPTI2UPC7LRGSQ2U7VGQQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 22:38:58.165272813 (535h9m58.036355848s) [noise=true , address=145.79.227.40:4051 , piece_id=7BJV5NHIVUPFKB5BLGSJXRDKWOKN43VLPTI2UPC7LRGSQ2U7VGQQ, seg_pos={0 0}, upload=469351]: Long tail cancellation due to optimal threshold Dec 27 22:38:58.165407931 (535h9m58.036490966s) [noise=true , address=45.91.116.11:4118 , piece_id=RJCXC4FLVAIHWSZGMCA324KIDU2QND2PZ4HUZUTUDSKSFUAS3PKQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 27 22:38:58.165428540 (535h9m58.036511575s) [noise=true , address=45.91.116.11:4118 , piece_id=RJCXC4FLVAIHWSZGMCA324KIDU2QND2PZ4HUZUTUDSKSFUAS3PKQ, seg_pos={0 0}, upload=469351]: Long tail cancellation due to optimal threshold Dec 27 22:38:58.165562365 (535h9m58.0366454s) [noise=true , address=77.90.31.11:4555 , piece_id=MXVALXCILZXUQ42LIUYGVQMJMBQOPQXD3KRFWP76DA6QUSLDJIQQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 22:38:58.165570010 (535h9m58.036653045s) [noise=true , address=77.90.31.11:4555 , piece_id=MXVALXCILZXUQ42LIUYGVQMJMBQOPQXD3KRFWP76DA6QUSLDJIQQ, seg_pos={0 0}, upload=469351]: Long tail cancellation due to optimal threshold Dec 27 22:38:58.165587112 (535h9m58.036670147s) [seg_pos={0 0}, upload=469351]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:38:58.165647487 (535h9m58.036730522s) [upload=469351]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:38:58.165651374 (535h9m58.036734409s) [upload=469351]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:38:58.457065586 (535h9m58.328148641s) [upload=469351]: Done uploading segments... Dec 27 22:38:58.457097867 (535h9m58.328180912s) [upload=469351]: Upload finished. err: <nil> Duration: 20.741137748s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................X............X.X...X.....X...X.] duration: 13.226386544s trace: https://jaeger.dev.storj.io/trace/1971cebe04e469c5 FAIL (2025-12-27 22:40:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 22:38:55.674939949 (535h9m55.546022984s) [upload=469351]: Starting upload Dec 27 22:38:55.675021665 (535h9m55.54610473s) [upload=469351]: Done starting upload Dec 27 22:38:55.675133347 (535h9m55.546216392s) [upload=469351]: Uploading segments... Dec 27 22:38:55.675243046 (535h9m55.546326081s) [upload=469351]: Got next segment. Inline: false Dec 27 22:38:55.675277822 (535h9m55.546360857s) [upload=469351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:38:55.675282120 (535h9m55.546365155s) [upload=469351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:38:55.915329712 (535h9m55.786412757s) [seg_pos={0 0}, upload=469351]: Begin upload segment... Dec 27 22:38:55.915375059 (535h9m55.786458104s) [seg_pos={0 0}, upload=469351]: Done begin upload segment. Dec 27 22:38:55.915400928 (535h9m55.786483973s) [noise=true , address=45.91.116.11:4118 , piece_id=RJCXC4FLVAIHWSZGMCA324KIDU2QND2PZ4HUZUTUDSKSFUAS3PKQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.915486932 (535h9m55.786569967s) [noise=true , address=77.90.31.14:4663 , piece_id=JCSKZEMHZPHENKILKR4BXB3IQ3DZV2GVROA3PYF26GMVQWC65ZRA, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.915954061 (535h9m55.787037106s) [noise=true , address=145.79.227.11:4729 , piece_id=YA3KINGOLGQHSRNTZBVVE42FUG4STIXNZZTQHWT7UXD6WP2ZTCDQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.916238022 (535h9m55.787321067s) [noise=true , address=45.91.116.5:4300 , piece_id=B3LAZ3VLK26ACE5O4OOF3FKNJKTL4SO5NZL6GTEHDX2RA22BCDXQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.916589411 (535h9m55.787672456s) [noise=true , address=145.79.227.14:4838 , piece_id=NE4MDMHA2T7T2QHKSQ5D7EY773EBQNRKE63PJG5DD2VLANQNQ6JA, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.916883972 (535h9m55.787967017s) [noise=true , address=45.91.116.13:4153 , piece_id=O3ARD6XVJOW4QWZZN3O7D74H6P2ZEFEKEGWWT6XAPP6UEYAY7UYQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.917175526 (535h9m55.788258561s) [noise=true , address=145.79.227.40:4051 , piece_id=7BJV5NHIVUPFKB5BLGSJXRDKWOKN43VLPTI2UPC7LRGSQ2U7VGQQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.917465209 (535h9m55.788548244s) [noise=true , address=145.79.227.46:4603 , piece_id=6XA47UBATOH5YKTUBSSYK4IML3YJQTTL3LTFP5LKYTJ3A273JX5A, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.917793784 (535h9m55.788876819s) [noise=true , address=145.79.227.13:4803 , piece_id=AFNTM5MXCJ6LJ6LPV7AMISNDYIL3AIRG7G3YJ4GYD3QJNRAKQVGQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.919206915 (535h9m55.79028995s) [noise=true , address=145.79.227.38:4978 , piece_id=MXONAWQCQ7XHL26GUTTWRB5NIO4VDTNUMTOBJUNIM77O3EMLRWKQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.920328821 (535h9m55.791411856s) [noise=true , address=145.79.227.18:4007 , piece_id=FNZAT4SEE5CQ2RAJWX2CTEDEHM3JPHNUJGEGSCPZ77YQZDLJIWGQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.921386004 (535h9m55.792469039s) [noise=true , address=145.79.227.2:4113 , piece_id=HNSWCV5G3DPBFVEIJYST4KGZPOSXEYYDMQJMUH4BEPZVUAR2ENRQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.922920596 (535h9m55.794003641s) [noise=true , address=145.79.227.4:4186 , piece_id=2A4YKAQMFNIRXSRG6MB762I57D2LSITKO4ANABPOLX5ZQGHUQNNQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.924434559 (535h9m55.795517604s) [noise=true , address=145.79.227.32:4836 , piece_id=C32FXTZYV6PKKHEEJTISNBXKY4YPZVWEZRXAMWHM4F75TMAWGEWA, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.925750875 (535h9m55.79683391s) [noise=true , address=145.79.227.21:4473 , piece_id=BJPDSF7BDNBUSBQITKBKW35OJB6YXLXCOM5JCWJXFAKYSAEQBLVQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.926548454 (535h9m55.797631499s) [noise=true , address=77.90.31.11:4555 , piece_id=MXVALXCILZXUQ42LIUYGVQMJMBQOPQXD3KRFWP76DA6QUSLDJIQQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:57.534201006 (535h9m57.405284041s) [upload=469351]: Next returned nil segment Dec 27 22:38:57.534220764 (535h9m57.405303799s) [upload=469351]: Waiting for error group managing segments... Dec 27 22:38:57.927837958 (535h9m57.798920993s) [noise=true , address=145.79.227.38:4978 , piece_id=MXONAWQCQ7XHL26GUTTWRB5NIO4VDTNUMTOBJUNIM77O3EMLRWKQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.928833213 (535h9m57.799916258s) [noise=true , address=145.79.227.14:4838 , piece_id=NE4MDMHA2T7T2QHKSQ5D7EY773EBQNRKE63PJG5DD2VLANQNQ6JA, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.933927804 (535h9m57.805010839s) [noise=true , address=145.79.227.21:4473 , piece_id=BJPDSF7BDNBUSBQITKBKW35OJB6YXLXCOM5JCWJXFAKYSAEQBLVQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.944054576 (535h9m57.815137611s) [noise=true , address=145.79.227.4:4186 , piece_id=2A4YKAQMFNIRXSRG6MB762I57D2LSITKO4ANABPOLX5ZQGHUQNNQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.945714578 (535h9m57.816797613s) [noise=true , address=145.79.227.18:4007 , piece_id=FNZAT4SEE5CQ2RAJWX2CTEDEHM3JPHNUJGEGSCPZ77YQZDLJIWGQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.946069714 (535h9m57.817152759s) [noise=true , address=145.79.227.11:4729 , piece_id=YA3KINGOLGQHSRNTZBVVE42FUG4STIXNZZTQHWT7UXD6WP2ZTCDQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.951466740 (535h9m57.822549776s) [noise=true , address=145.79.227.46:4603 , piece_id=6XA47UBATOH5YKTUBSSYK4IML3YJQTTL3LTFP5LKYTJ3A273JX5A, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.951553035 (535h9m57.82263608s) [noise=true , address=145.79.227.32:4836 , piece_id=C32FXTZYV6PKKHEEJTISNBXKY4YPZVWEZRXAMWHM4F75TMAWGEWA, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.951902110 (535h9m57.822985145s) [noise=true , address=145.79.227.13:4803 , piece_id=AFNTM5MXCJ6LJ6LPV7AMISNDYIL3AIRG7G3YJ4GYD3QJNRAKQVGQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.957315117 (535h9m57.828398152s) [noise=true , address=145.79.227.2:4113 , piece_id=HNSWCV5G3DPBFVEIJYST4KGZPOSXEYYDMQJMUH4BEPZVUAR2ENRQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.121309660 (535h9m57.992392695s) [noise=true , address=77.90.31.14:4663 , piece_id=JCSKZEMHZPHENKILKR4BXB3IQ3DZV2GVROA3PYF26GMVQWC65ZRA, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.137653071 (535h9m58.008736106s) [noise=true , address=45.91.116.13:4153 , piece_id=O3ARD6XVJOW4QWZZN3O7D74H6P2ZEFEKEGWWT6XAPP6UEYAY7UYQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.165074426 (535h9m58.036157461s) [noise=true , address=45.91.116.5:4300 , piece_id=B3LAZ3VLK26ACE5O4OOF3FKNJKTL4SO5NZL6GTEHDX2RA22BCDXQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.165093021 (535h9m58.036176056s) [seg_pos={0 0}, upload=469351]: Segment reached optimal threshold of 13 pieces. Dec 27 22:38:58.165266581 (535h9m58.036349617s) [noise=true , address=145.79.227.40:4051 , piece_id=7BJV5NHIVUPFKB5BLGSJXRDKWOKN43VLPTI2UPC7LRGSQ2U7VGQQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 22:38:58.165272813 (535h9m58.036355848s) [noise=true , address=145.79.227.40:4051 , piece_id=7BJV5NHIVUPFKB5BLGSJXRDKWOKN43VLPTI2UPC7LRGSQ2U7VGQQ, seg_pos={0 0}, upload=469351]: Long tail cancellation due to optimal threshold Dec 27 22:38:58.165407931 (535h9m58.036490966s) [noise=true , address=45.91.116.11:4118 , piece_id=RJCXC4FLVAIHWSZGMCA324KIDU2QND2PZ4HUZUTUDSKSFUAS3PKQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 27 22:38:58.165428540 (535h9m58.036511575s) [noise=true , address=45.91.116.11:4118 , piece_id=RJCXC4FLVAIHWSZGMCA324KIDU2QND2PZ4HUZUTUDSKSFUAS3PKQ, seg_pos={0 0}, upload=469351]: Long tail cancellation due to optimal threshold Dec 27 22:38:58.165562365 (535h9m58.0366454s) [noise=true , address=77.90.31.11:4555 , piece_id=MXVALXCILZXUQ42LIUYGVQMJMBQOPQXD3KRFWP76DA6QUSLDJIQQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 22:38:58.165570010 (535h9m58.036653045s) [noise=true , address=77.90.31.11:4555 , piece_id=MXVALXCILZXUQ42LIUYGVQMJMBQOPQXD3KRFWP76DA6QUSLDJIQQ, seg_pos={0 0}, upload=469351]: Long tail cancellation due to optimal threshold Dec 27 22:38:58.165587112 (535h9m58.036670147s) [seg_pos={0 0}, upload=469351]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:38:58.165647487 (535h9m58.036730522s) [upload=469351]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:38:58.165651374 (535h9m58.036734409s) [upload=469351]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:38:58.457065586 (535h9m58.328148641s) [upload=469351]: Done uploading segments... Dec 27 22:38:58.457097867 (535h9m58.328180912s) [upload=469351]: Upload finished. err: <nil> Duration: 13.226449293s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 153.079539ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:39:56 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 153.106099ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................X............X.X...X.....X...X] duration: 30.002895287s trace: https://jaeger.dev.storj.io/trace/1971cebe04e469c5 FAIL (2025-12-27 22:39:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 22:38:55.674939949 (535h9m55.546022984s) [upload=469351]: Starting upload Dec 27 22:38:55.675021665 (535h9m55.54610473s) [upload=469351]: Done starting upload Dec 27 22:38:55.675133347 (535h9m55.546216392s) [upload=469351]: Uploading segments... Dec 27 22:38:55.675243046 (535h9m55.546326081s) [upload=469351]: Got next segment. Inline: false Dec 27 22:38:55.675277822 (535h9m55.546360857s) [upload=469351]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:38:55.675282120 (535h9m55.546365155s) [upload=469351]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:38:55.915329712 (535h9m55.786412757s) [seg_pos={0 0}, upload=469351]: Begin upload segment... Dec 27 22:38:55.915375059 (535h9m55.786458104s) [seg_pos={0 0}, upload=469351]: Done begin upload segment. Dec 27 22:38:55.915400928 (535h9m55.786483973s) [noise=true , address=45.91.116.11:4118 , piece_id=RJCXC4FLVAIHWSZGMCA324KIDU2QND2PZ4HUZUTUDSKSFUAS3PKQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.915486932 (535h9m55.786569967s) [noise=true , address=77.90.31.14:4663 , piece_id=JCSKZEMHZPHENKILKR4BXB3IQ3DZV2GVROA3PYF26GMVQWC65ZRA, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.915954061 (535h9m55.787037106s) [noise=true , address=145.79.227.11:4729 , piece_id=YA3KINGOLGQHSRNTZBVVE42FUG4STIXNZZTQHWT7UXD6WP2ZTCDQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.916238022 (535h9m55.787321067s) [noise=true , address=45.91.116.5:4300 , piece_id=B3LAZ3VLK26ACE5O4OOF3FKNJKTL4SO5NZL6GTEHDX2RA22BCDXQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.916589411 (535h9m55.787672456s) [noise=true , address=145.79.227.14:4838 , piece_id=NE4MDMHA2T7T2QHKSQ5D7EY773EBQNRKE63PJG5DD2VLANQNQ6JA, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.916883972 (535h9m55.787967017s) [noise=true , address=45.91.116.13:4153 , piece_id=O3ARD6XVJOW4QWZZN3O7D74H6P2ZEFEKEGWWT6XAPP6UEYAY7UYQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.917175526 (535h9m55.788258561s) [noise=true , address=145.79.227.40:4051 , piece_id=7BJV5NHIVUPFKB5BLGSJXRDKWOKN43VLPTI2UPC7LRGSQ2U7VGQQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.917465209 (535h9m55.788548244s) [noise=true , address=145.79.227.46:4603 , piece_id=6XA47UBATOH5YKTUBSSYK4IML3YJQTTL3LTFP5LKYTJ3A273JX5A, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.917793784 (535h9m55.788876819s) [noise=true , address=145.79.227.13:4803 , piece_id=AFNTM5MXCJ6LJ6LPV7AMISNDYIL3AIRG7G3YJ4GYD3QJNRAKQVGQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.919206915 (535h9m55.79028995s) [noise=true , address=145.79.227.38:4978 , piece_id=MXONAWQCQ7XHL26GUTTWRB5NIO4VDTNUMTOBJUNIM77O3EMLRWKQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.920328821 (535h9m55.791411856s) [noise=true , address=145.79.227.18:4007 , piece_id=FNZAT4SEE5CQ2RAJWX2CTEDEHM3JPHNUJGEGSCPZ77YQZDLJIWGQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.921386004 (535h9m55.792469039s) [noise=true , address=145.79.227.2:4113 , piece_id=HNSWCV5G3DPBFVEIJYST4KGZPOSXEYYDMQJMUH4BEPZVUAR2ENRQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.922920596 (535h9m55.794003641s) [noise=true , address=145.79.227.4:4186 , piece_id=2A4YKAQMFNIRXSRG6MB762I57D2LSITKO4ANABPOLX5ZQGHUQNNQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.924434559 (535h9m55.795517604s) [noise=true , address=145.79.227.32:4836 , piece_id=C32FXTZYV6PKKHEEJTISNBXKY4YPZVWEZRXAMWHM4F75TMAWGEWA, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.925750875 (535h9m55.79683391s) [noise=true , address=145.79.227.21:4473 , piece_id=BJPDSF7BDNBUSBQITKBKW35OJB6YXLXCOM5JCWJXFAKYSAEQBLVQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:55.926548454 (535h9m55.797631499s) [noise=true , address=77.90.31.11:4555 , piece_id=MXVALXCILZXUQ42LIUYGVQMJMBQOPQXD3KRFWP76DA6QUSLDJIQQ, seg_pos={0 0}, upload=469351]: Uploading piece... Dec 27 22:38:57.534201006 (535h9m57.405284041s) [upload=469351]: Next returned nil segment Dec 27 22:38:57.534220764 (535h9m57.405303799s) [upload=469351]: Waiting for error group managing segments... Dec 27 22:38:57.927837958 (535h9m57.798920993s) [noise=true , address=145.79.227.38:4978 , piece_id=MXONAWQCQ7XHL26GUTTWRB5NIO4VDTNUMTOBJUNIM77O3EMLRWKQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.928833213 (535h9m57.799916258s) [noise=true , address=145.79.227.14:4838 , piece_id=NE4MDMHA2T7T2QHKSQ5D7EY773EBQNRKE63PJG5DD2VLANQNQ6JA, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.933927804 (535h9m57.805010839s) [noise=true , address=145.79.227.21:4473 , piece_id=BJPDSF7BDNBUSBQITKBKW35OJB6YXLXCOM5JCWJXFAKYSAEQBLVQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.944054576 (535h9m57.815137611s) [noise=true , address=145.79.227.4:4186 , piece_id=2A4YKAQMFNIRXSRG6MB762I57D2LSITKO4ANABPOLX5ZQGHUQNNQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.945714578 (535h9m57.816797613s) [noise=true , address=145.79.227.18:4007 , piece_id=FNZAT4SEE5CQ2RAJWX2CTEDEHM3JPHNUJGEGSCPZ77YQZDLJIWGQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.946069714 (535h9m57.817152759s) [noise=true , address=145.79.227.11:4729 , piece_id=YA3KINGOLGQHSRNTZBVVE42FUG4STIXNZZTQHWT7UXD6WP2ZTCDQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.951466740 (535h9m57.822549776s) [noise=true , address=145.79.227.46:4603 , piece_id=6XA47UBATOH5YKTUBSSYK4IML3YJQTTL3LTFP5LKYTJ3A273JX5A, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.951553035 (535h9m57.82263608s) [noise=true , address=145.79.227.32:4836 , piece_id=C32FXTZYV6PKKHEEJTISNBXKY4YPZVWEZRXAMWHM4F75TMAWGEWA, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.951902110 (535h9m57.822985145s) [noise=true , address=145.79.227.13:4803 , piece_id=AFNTM5MXCJ6LJ6LPV7AMISNDYIL3AIRG7G3YJ4GYD3QJNRAKQVGQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:57.957315117 (535h9m57.828398152s) [noise=true , address=145.79.227.2:4113 , piece_id=HNSWCV5G3DPBFVEIJYST4KGZPOSXEYYDMQJMUH4BEPZVUAR2ENRQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.121309660 (535h9m57.992392695s) [noise=true , address=77.90.31.14:4663 , piece_id=JCSKZEMHZPHENKILKR4BXB3IQ3DZV2GVROA3PYF26GMVQWC65ZRA, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.137653071 (535h9m58.008736106s) [noise=true , address=45.91.116.13:4153 , piece_id=O3ARD6XVJOW4QWZZN3O7D74H6P2ZEFEKEGWWT6XAPP6UEYAY7UYQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.165074426 (535h9m58.036157461s) [noise=true , address=45.91.116.5:4300 , piece_id=B3LAZ3VLK26ACE5O4OOF3FKNJKTL4SO5NZL6GTEHDX2RA22BCDXQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: <nil> Dec 27 22:38:58.165093021 (535h9m58.036176056s) [seg_pos={0 0}, upload=469351]: Segment reached optimal threshold of 13 pieces. Dec 27 22:38:58.165266581 (535h9m58.036349617s) [noise=true , address=145.79.227.40:4051 , piece_id=7BJV5NHIVUPFKB5BLGSJXRDKWOKN43VLPTI2UPC7LRGSQ2U7VGQQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 22:38:58.165272813 (535h9m58.036355848s) [noise=true , address=145.79.227.40:4051 , piece_id=7BJV5NHIVUPFKB5BLGSJXRDKWOKN43VLPTI2UPC7LRGSQ2U7VGQQ, seg_pos={0 0}, upload=469351]: Long tail cancellation due to optimal threshold Dec 27 22:38:58.165407931 (535h9m58.036490966s) [noise=true , address=45.91.116.11:4118 , piece_id=RJCXC4FLVAIHWSZGMCA324KIDU2QND2PZ4HUZUTUDSKSFUAS3PKQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 27 22:38:58.165428540 (535h9m58.036511575s) [noise=true , address=45.91.116.11:4118 , piece_id=RJCXC4FLVAIHWSZGMCA324KIDU2QND2PZ4HUZUTUDSKSFUAS3PKQ, seg_pos={0 0}, upload=469351]: Long tail cancellation due to optimal threshold Dec 27 22:38:58.165562365 (535h9m58.0366454s) [noise=true , address=77.90.31.11:4555 , piece_id=MXVALXCILZXUQ42LIUYGVQMJMBQOPQXD3KRFWP76DA6QUSLDJIQQ, seg_pos={0 0}, upload=469351]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 27 22:38:58.165570010 (535h9m58.036653045s) [noise=true , address=77.90.31.11:4555 , piece_id=MXVALXCILZXUQ42LIUYGVQMJMBQOPQXD3KRFWP76DA6QUSLDJIQQ, seg_pos={0 0}, upload=469351]: Long tail cancellation due to optimal threshold Dec 27 22:38:58.165587112 (535h9m58.036670147s) [seg_pos={0 0}, upload=469351]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:38:58.165647487 (535h9m58.036730522s) [upload=469351]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:38:58.165651374 (535h9m58.036734409s) [upload=469351]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:38:58.457065586 (535h9m58.328148641s) [upload=469351]: Done uploading segments... Dec 27 22:38:58.457097867 (535h9m58.328180912s) [upload=469351]: Upload finished. err: <nil> Duration: 30.002918191s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 158.707196ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:39:04 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 158.74108ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 156.371337ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:38:21 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 156.396816ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................X............X.X...X.....X...] duration: 6.066815963s trace: https://jaeger.dev.storj.io/trace/7eeb6cfd859a0038 FAIL (2025-12-27 22:38:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q) context deadline exceeded ======LOGS====== Dec 27 22:35:36.307110178 (535h6m36.178193213s) [upload=469300]: Starting upload Dec 27 22:35:36.307180792 (535h6m36.178263837s) [upload=469300]: Done starting upload Dec 27 22:35:36.307286814 (535h6m36.178369859s) [upload=469300]: Uploading segments... Dec 27 22:35:36.307425117 (535h6m36.178508162s) [upload=469300]: Got next segment. Inline: false Dec 27 22:35:36.307476645 (535h6m36.17855968s) [upload=469300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:35:36.307485442 (535h6m36.178568478s) [upload=469300]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:35:36.550202498 (535h6m36.421285533s) [seg_pos={0 0}, upload=469300]: Begin upload segment... Dec 27 22:35:36.550262552 (535h6m36.421345587s) [seg_pos={0 0}, upload=469300]: Done begin upload segment. Dec 27 22:35:36.550294012 (535h6m36.421377067s) [noise=true , address=145.79.227.7:4293 , piece_id=D2XC5E5SDN3NXRQ2Q3QB7PNULJ744WYEQNZG7RNLEWWLAGB2N3HQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550303389 (535h6m36.421386424s) [noise=true , address=145.79.227.36:4519 , piece_id=GLGX3ROM4Y4QEIENFKNJPLGL47FKXTQLKSBKZTBMULIOTQ27DULQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550352904 (535h6m36.421435939s) [noise=true , address=145.79.227.11:4730 , piece_id=JQM2K2K4OEDGIL4VOOKGMKDWYKT4UADVWTPEL4HXYTLLXRJUIOOQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550959860 (535h6m36.422042895s) [noise=true , address=145.79.227.12:4765 , piece_id=Y2PZSZUPEXQHXZ7WXO3DGB2JKB2ADX4ZSKCKXF3MNDYTER6XQ6FA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.551337489 (535h6m36.422420534s) [noise=true , address=145.79.227.2:4115 , piece_id=QKYO6ZCJYVBFM6EB6TCCA5P6USGV46MC7CXS7HZSHPYHOIJUY52Q, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.551702815 (535h6m36.42278586s) [noise=true , address=145.79.227.17:4970 , piece_id=UAR77FFRTYRR2VZSMAV4HGXCXDFS6IOEABMWCPX4P5GX3F7L4MHA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.552073340 (535h6m36.423156395s) [noise=true , address=145.79.227.45:4229 , piece_id=PWCS73FOEMLUMAOCYE22YIMJR6EFVR6H44KZ7HBGUUICMQJRMK3A, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.552462371 (535h6m36.423545407s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.553003401 (535h6m36.424086446s) [noise=true , address=77.90.31.13:4627 , piece_id=BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.553460722 (535h6m36.424543767s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554153581 (535h6m36.425236626s) [noise=true , address=145.79.227.4:4185 , piece_id=GLR3OXYB7YYBELFNB6JGXEOITJPC2NE7RO6DSXWGNJ4KD45FHVRA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554610161 (535h6m36.425693196s) [noise=true , address=45.91.116.5:4299 , piece_id=4SRWRWNGVZCBJA52LHJDIGZFX4KQYVKUJHJDBQF4NWHTPSJRH4LA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554625260 (535h6m36.425708295s) [noise=true , address=45.91.116.13:4153 , piece_id=N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555190296 (535h6m36.426273331s) [noise=true , address=145.79.227.10:4402 , piece_id=AY5DXAWIF6HLGOTDS7HAUC2P6OZK4GBBYRWAKJSE3GCQ4NXPNVSA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555281950 (535h6m36.426364985s) [noise=true , address=145.79.227.52:4446 , piece_id=VGATGH2KIKLQINJPDSNGAXYPMXJMGQVROFP5VCKBI7LYCZIDYLFA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555890368 (535h6m36.426973414s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:38.277154486 (535h6m38.148237531s) [upload=469300]: Next returned nil segment Dec 27 22:35:38.277168422 (535h6m38.148251457s) [upload=469300]: Waiting for error group managing segments... Dec 27 22:35:38.555483832 (535h6m38.426566867s) [noise=true , address=145.79.227.11:4730 , piece_id=JQM2K2K4OEDGIL4VOOKGMKDWYKT4UADVWTPEL4HXYTLLXRJUIOOQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.559624757 (535h6m38.430707792s) [noise=true , address=145.79.227.10:4402 , piece_id=AY5DXAWIF6HLGOTDS7HAUC2P6OZK4GBBYRWAKJSE3GCQ4NXPNVSA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.560650140 (535h6m38.431733175s) [noise=true , address=145.79.227.17:4970 , piece_id=UAR77FFRTYRR2VZSMAV4HGXCXDFS6IOEABMWCPX4P5GX3F7L4MHA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.564909921 (535h6m38.435992986s) [noise=true , address=145.79.227.7:4293 , piece_id=D2XC5E5SDN3NXRQ2Q3QB7PNULJ744WYEQNZG7RNLEWWLAGB2N3HQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.572882863 (535h6m38.443965898s) [noise=true , address=145.79.227.45:4229 , piece_id=PWCS73FOEMLUMAOCYE22YIMJR6EFVR6H44KZ7HBGUUICMQJRMK3A, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.573745586 (535h6m38.444828631s) [noise=true , address=145.79.227.4:4185 , piece_id=GLR3OXYB7YYBELFNB6JGXEOITJPC2NE7RO6DSXWGNJ4KD45FHVRA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.574549066 (535h6m38.445632101s) [noise=true , address=145.79.227.36:4519 , piece_id=GLGX3ROM4Y4QEIENFKNJPLGL47FKXTQLKSBKZTBMULIOTQ27DULQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.574760709 (535h6m38.445843744s) [noise=true , address=145.79.227.2:4115 , piece_id=QKYO6ZCJYVBFM6EB6TCCA5P6USGV46MC7CXS7HZSHPYHOIJUY52Q, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.581427843 (535h6m38.452510888s) [noise=true , address=145.79.227.52:4446 , piece_id=VGATGH2KIKLQINJPDSNGAXYPMXJMGQVROFP5VCKBI7LYCZIDYLFA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.757074013 (535h6m38.628157048s) [noise=true , address=145.79.227.12:4765 , piece_id=Y2PZSZUPEXQHXZ7WXO3DGB2JKB2ADX4ZSKCKXF3MNDYTER6XQ6FA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.769644851 (535h6m38.640727886s) [noise=true , address=45.91.116.5:4299 , piece_id=4SRWRWNGVZCBJA52LHJDIGZFX4KQYVKUJHJDBQF4NWHTPSJRH4LA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.814953196 (535h6m38.686036231s) [noise=true , address=45.91.116.13:4153 , piece_id=N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.833765998 (535h6m38.704849033s) [noise=true , address=77.90.31.13:4627 , piece_id=BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.833822626 (535h6m38.704905661s) [seg_pos={0 0}, upload=469300]: Segment reached optimal threshold of 13 pieces. Dec 27 22:35:38.834139189 (535h6m38.705222224s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 22:35:38.834148266 (535h6m38.705231301s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834306488 (535h6m38.705389523s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection Dec 27 22:35:38.834321235 (535h6m38.70540427s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834480158 (535h6m38.705563193s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 22:35:38.834486290 (535h6m38.705569335s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834518411 (535h6m38.705601456s) [seg_pos={0 0}, upload=469300]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:35:38.834595878 (535h6m38.705678913s) [upload=469300]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:35:38.834599966 (535h6m38.705683001s) [upload=469300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:35:39.130055424 (535h6m39.001138469s) [upload=469300]: Done uploading segments... Dec 27 22:35:39.130079990 (535h6m39.001163035s) [upload=469300]: Upload finished. err: <nil> Duration: 6.066849526s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 168.335068ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:37:38 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 168.370015ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................X............X.X...X.....X..] duration: 9.511020708s trace: https://jaeger.dev.storj.io/trace/7eeb6cfd859a0038 FAIL (2025-12-27 22:37:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q) context deadline exceeded ======LOGS====== Dec 27 22:35:36.307110178 (535h6m36.178193213s) [upload=469300]: Starting upload Dec 27 22:35:36.307180792 (535h6m36.178263837s) [upload=469300]: Done starting upload Dec 27 22:35:36.307286814 (535h6m36.178369859s) [upload=469300]: Uploading segments... Dec 27 22:35:36.307425117 (535h6m36.178508162s) [upload=469300]: Got next segment. Inline: false Dec 27 22:35:36.307476645 (535h6m36.17855968s) [upload=469300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:35:36.307485442 (535h6m36.178568478s) [upload=469300]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:35:36.550202498 (535h6m36.421285533s) [seg_pos={0 0}, upload=469300]: Begin upload segment... Dec 27 22:35:36.550262552 (535h6m36.421345587s) [seg_pos={0 0}, upload=469300]: Done begin upload segment. Dec 27 22:35:36.550294012 (535h6m36.421377067s) [noise=true , address=145.79.227.7:4293 , piece_id=D2XC5E5SDN3NXRQ2Q3QB7PNULJ744WYEQNZG7RNLEWWLAGB2N3HQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550303389 (535h6m36.421386424s) [noise=true , address=145.79.227.36:4519 , piece_id=GLGX3ROM4Y4QEIENFKNJPLGL47FKXTQLKSBKZTBMULIOTQ27DULQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550352904 (535h6m36.421435939s) [noise=true , address=145.79.227.11:4730 , piece_id=JQM2K2K4OEDGIL4VOOKGMKDWYKT4UADVWTPEL4HXYTLLXRJUIOOQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550959860 (535h6m36.422042895s) [noise=true , address=145.79.227.12:4765 , piece_id=Y2PZSZUPEXQHXZ7WXO3DGB2JKB2ADX4ZSKCKXF3MNDYTER6XQ6FA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.551337489 (535h6m36.422420534s) [noise=true , address=145.79.227.2:4115 , piece_id=QKYO6ZCJYVBFM6EB6TCCA5P6USGV46MC7CXS7HZSHPYHOIJUY52Q, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.551702815 (535h6m36.42278586s) [noise=true , address=145.79.227.17:4970 , piece_id=UAR77FFRTYRR2VZSMAV4HGXCXDFS6IOEABMWCPX4P5GX3F7L4MHA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.552073340 (535h6m36.423156395s) [noise=true , address=145.79.227.45:4229 , piece_id=PWCS73FOEMLUMAOCYE22YIMJR6EFVR6H44KZ7HBGUUICMQJRMK3A, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.552462371 (535h6m36.423545407s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.553003401 (535h6m36.424086446s) [noise=true , address=77.90.31.13:4627 , piece_id=BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.553460722 (535h6m36.424543767s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554153581 (535h6m36.425236626s) [noise=true , address=145.79.227.4:4185 , piece_id=GLR3OXYB7YYBELFNB6JGXEOITJPC2NE7RO6DSXWGNJ4KD45FHVRA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554610161 (535h6m36.425693196s) [noise=true , address=45.91.116.5:4299 , piece_id=4SRWRWNGVZCBJA52LHJDIGZFX4KQYVKUJHJDBQF4NWHTPSJRH4LA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554625260 (535h6m36.425708295s) [noise=true , address=45.91.116.13:4153 , piece_id=N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555190296 (535h6m36.426273331s) [noise=true , address=145.79.227.10:4402 , piece_id=AY5DXAWIF6HLGOTDS7HAUC2P6OZK4GBBYRWAKJSE3GCQ4NXPNVSA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555281950 (535h6m36.426364985s) [noise=true , address=145.79.227.52:4446 , piece_id=VGATGH2KIKLQINJPDSNGAXYPMXJMGQVROFP5VCKBI7LYCZIDYLFA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555890368 (535h6m36.426973414s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:38.277154486 (535h6m38.148237531s) [upload=469300]: Next returned nil segment Dec 27 22:35:38.277168422 (535h6m38.148251457s) [upload=469300]: Waiting for error group managing segments... Dec 27 22:35:38.555483832 (535h6m38.426566867s) [noise=true , address=145.79.227.11:4730 , piece_id=JQM2K2K4OEDGIL4VOOKGMKDWYKT4UADVWTPEL4HXYTLLXRJUIOOQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.559624757 (535h6m38.430707792s) [noise=true , address=145.79.227.10:4402 , piece_id=AY5DXAWIF6HLGOTDS7HAUC2P6OZK4GBBYRWAKJSE3GCQ4NXPNVSA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.560650140 (535h6m38.431733175s) [noise=true , address=145.79.227.17:4970 , piece_id=UAR77FFRTYRR2VZSMAV4HGXCXDFS6IOEABMWCPX4P5GX3F7L4MHA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.564909921 (535h6m38.435992986s) [noise=true , address=145.79.227.7:4293 , piece_id=D2XC5E5SDN3NXRQ2Q3QB7PNULJ744WYEQNZG7RNLEWWLAGB2N3HQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.572882863 (535h6m38.443965898s) [noise=true , address=145.79.227.45:4229 , piece_id=PWCS73FOEMLUMAOCYE22YIMJR6EFVR6H44KZ7HBGUUICMQJRMK3A, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.573745586 (535h6m38.444828631s) [noise=true , address=145.79.227.4:4185 , piece_id=GLR3OXYB7YYBELFNB6JGXEOITJPC2NE7RO6DSXWGNJ4KD45FHVRA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.574549066 (535h6m38.445632101s) [noise=true , address=145.79.227.36:4519 , piece_id=GLGX3ROM4Y4QEIENFKNJPLGL47FKXTQLKSBKZTBMULIOTQ27DULQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.574760709 (535h6m38.445843744s) [noise=true , address=145.79.227.2:4115 , piece_id=QKYO6ZCJYVBFM6EB6TCCA5P6USGV46MC7CXS7HZSHPYHOIJUY52Q, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.581427843 (535h6m38.452510888s) [noise=true , address=145.79.227.52:4446 , piece_id=VGATGH2KIKLQINJPDSNGAXYPMXJMGQVROFP5VCKBI7LYCZIDYLFA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.757074013 (535h6m38.628157048s) [noise=true , address=145.79.227.12:4765 , piece_id=Y2PZSZUPEXQHXZ7WXO3DGB2JKB2ADX4ZSKCKXF3MNDYTER6XQ6FA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.769644851 (535h6m38.640727886s) [noise=true , address=45.91.116.5:4299 , piece_id=4SRWRWNGVZCBJA52LHJDIGZFX4KQYVKUJHJDBQF4NWHTPSJRH4LA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.814953196 (535h6m38.686036231s) [noise=true , address=45.91.116.13:4153 , piece_id=N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.833765998 (535h6m38.704849033s) [noise=true , address=77.90.31.13:4627 , piece_id=BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.833822626 (535h6m38.704905661s) [seg_pos={0 0}, upload=469300]: Segment reached optimal threshold of 13 pieces. Dec 27 22:35:38.834139189 (535h6m38.705222224s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 22:35:38.834148266 (535h6m38.705231301s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834306488 (535h6m38.705389523s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection Dec 27 22:35:38.834321235 (535h6m38.70540427s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834480158 (535h6m38.705563193s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 22:35:38.834486290 (535h6m38.705569335s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834518411 (535h6m38.705601456s) [seg_pos={0 0}, upload=469300]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:35:38.834595878 (535h6m38.705678913s) [upload=469300]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:35:38.834599966 (535h6m38.705683001s) [upload=469300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:35:39.130055424 (535h6m39.001138469s) [upload=469300]: Done uploading segments... Dec 27 22:35:39.130079990 (535h6m39.001163035s) [upload=469300]: Upload finished. err: <nil> Duration: 9.511053319s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 165.44756ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:37:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 165.476685ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X............X.X...X.....X.] duration: 15.859748376s trace: https://jaeger.dev.storj.io/trace/7eeb6cfd859a0038 FAIL (2025-12-27 22:36:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q) context deadline exceeded ======LOGS====== Dec 27 22:35:36.307110178 (535h6m36.178193213s) [upload=469300]: Starting upload Dec 27 22:35:36.307180792 (535h6m36.178263837s) [upload=469300]: Done starting upload Dec 27 22:35:36.307286814 (535h6m36.178369859s) [upload=469300]: Uploading segments... Dec 27 22:35:36.307425117 (535h6m36.178508162s) [upload=469300]: Got next segment. Inline: false Dec 27 22:35:36.307476645 (535h6m36.17855968s) [upload=469300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:35:36.307485442 (535h6m36.178568478s) [upload=469300]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:35:36.550202498 (535h6m36.421285533s) [seg_pos={0 0}, upload=469300]: Begin upload segment... Dec 27 22:35:36.550262552 (535h6m36.421345587s) [seg_pos={0 0}, upload=469300]: Done begin upload segment. Dec 27 22:35:36.550294012 (535h6m36.421377067s) [noise=true , address=145.79.227.7:4293 , piece_id=D2XC5E5SDN3NXRQ2Q3QB7PNULJ744WYEQNZG7RNLEWWLAGB2N3HQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550303389 (535h6m36.421386424s) [noise=true , address=145.79.227.36:4519 , piece_id=GLGX3ROM4Y4QEIENFKNJPLGL47FKXTQLKSBKZTBMULIOTQ27DULQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550352904 (535h6m36.421435939s) [noise=true , address=145.79.227.11:4730 , piece_id=JQM2K2K4OEDGIL4VOOKGMKDWYKT4UADVWTPEL4HXYTLLXRJUIOOQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550959860 (535h6m36.422042895s) [noise=true , address=145.79.227.12:4765 , piece_id=Y2PZSZUPEXQHXZ7WXO3DGB2JKB2ADX4ZSKCKXF3MNDYTER6XQ6FA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.551337489 (535h6m36.422420534s) [noise=true , address=145.79.227.2:4115 , piece_id=QKYO6ZCJYVBFM6EB6TCCA5P6USGV46MC7CXS7HZSHPYHOIJUY52Q, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.551702815 (535h6m36.42278586s) [noise=true , address=145.79.227.17:4970 , piece_id=UAR77FFRTYRR2VZSMAV4HGXCXDFS6IOEABMWCPX4P5GX3F7L4MHA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.552073340 (535h6m36.423156395s) [noise=true , address=145.79.227.45:4229 , piece_id=PWCS73FOEMLUMAOCYE22YIMJR6EFVR6H44KZ7HBGUUICMQJRMK3A, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.552462371 (535h6m36.423545407s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.553003401 (535h6m36.424086446s) [noise=true , address=77.90.31.13:4627 , piece_id=BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.553460722 (535h6m36.424543767s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554153581 (535h6m36.425236626s) [noise=true , address=145.79.227.4:4185 , piece_id=GLR3OXYB7YYBELFNB6JGXEOITJPC2NE7RO6DSXWGNJ4KD45FHVRA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554610161 (535h6m36.425693196s) [noise=true , address=45.91.116.5:4299 , piece_id=4SRWRWNGVZCBJA52LHJDIGZFX4KQYVKUJHJDBQF4NWHTPSJRH4LA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554625260 (535h6m36.425708295s) [noise=true , address=45.91.116.13:4153 , piece_id=N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555190296 (535h6m36.426273331s) [noise=true , address=145.79.227.10:4402 , piece_id=AY5DXAWIF6HLGOTDS7HAUC2P6OZK4GBBYRWAKJSE3GCQ4NXPNVSA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555281950 (535h6m36.426364985s) [noise=true , address=145.79.227.52:4446 , piece_id=VGATGH2KIKLQINJPDSNGAXYPMXJMGQVROFP5VCKBI7LYCZIDYLFA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555890368 (535h6m36.426973414s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:38.277154486 (535h6m38.148237531s) [upload=469300]: Next returned nil segment Dec 27 22:35:38.277168422 (535h6m38.148251457s) [upload=469300]: Waiting for error group managing segments... Dec 27 22:35:38.555483832 (535h6m38.426566867s) [noise=true , address=145.79.227.11:4730 , piece_id=JQM2K2K4OEDGIL4VOOKGMKDWYKT4UADVWTPEL4HXYTLLXRJUIOOQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.559624757 (535h6m38.430707792s) [noise=true , address=145.79.227.10:4402 , piece_id=AY5DXAWIF6HLGOTDS7HAUC2P6OZK4GBBYRWAKJSE3GCQ4NXPNVSA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.560650140 (535h6m38.431733175s) [noise=true , address=145.79.227.17:4970 , piece_id=UAR77FFRTYRR2VZSMAV4HGXCXDFS6IOEABMWCPX4P5GX3F7L4MHA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.564909921 (535h6m38.435992986s) [noise=true , address=145.79.227.7:4293 , piece_id=D2XC5E5SDN3NXRQ2Q3QB7PNULJ744WYEQNZG7RNLEWWLAGB2N3HQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.572882863 (535h6m38.443965898s) [noise=true , address=145.79.227.45:4229 , piece_id=PWCS73FOEMLUMAOCYE22YIMJR6EFVR6H44KZ7HBGUUICMQJRMK3A, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.573745586 (535h6m38.444828631s) [noise=true , address=145.79.227.4:4185 , piece_id=GLR3OXYB7YYBELFNB6JGXEOITJPC2NE7RO6DSXWGNJ4KD45FHVRA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.574549066 (535h6m38.445632101s) [noise=true , address=145.79.227.36:4519 , piece_id=GLGX3ROM4Y4QEIENFKNJPLGL47FKXTQLKSBKZTBMULIOTQ27DULQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.574760709 (535h6m38.445843744s) [noise=true , address=145.79.227.2:4115 , piece_id=QKYO6ZCJYVBFM6EB6TCCA5P6USGV46MC7CXS7HZSHPYHOIJUY52Q, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.581427843 (535h6m38.452510888s) [noise=true , address=145.79.227.52:4446 , piece_id=VGATGH2KIKLQINJPDSNGAXYPMXJMGQVROFP5VCKBI7LYCZIDYLFA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.757074013 (535h6m38.628157048s) [noise=true , address=145.79.227.12:4765 , piece_id=Y2PZSZUPEXQHXZ7WXO3DGB2JKB2ADX4ZSKCKXF3MNDYTER6XQ6FA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.769644851 (535h6m38.640727886s) [noise=true , address=45.91.116.5:4299 , piece_id=4SRWRWNGVZCBJA52LHJDIGZFX4KQYVKUJHJDBQF4NWHTPSJRH4LA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.814953196 (535h6m38.686036231s) [noise=true , address=45.91.116.13:4153 , piece_id=N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.833765998 (535h6m38.704849033s) [noise=true , address=77.90.31.13:4627 , piece_id=BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.833822626 (535h6m38.704905661s) [seg_pos={0 0}, upload=469300]: Segment reached optimal threshold of 13 pieces. Dec 27 22:35:38.834139189 (535h6m38.705222224s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 22:35:38.834148266 (535h6m38.705231301s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834306488 (535h6m38.705389523s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection Dec 27 22:35:38.834321235 (535h6m38.70540427s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834480158 (535h6m38.705563193s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 22:35:38.834486290 (535h6m38.705569335s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834518411 (535h6m38.705601456s) [seg_pos={0 0}, upload=469300]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:35:38.834595878 (535h6m38.705678913s) [upload=469300]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:35:38.834599966 (535h6m38.705683001s) [upload=469300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:35:39.130055424 (535h6m39.001138469s) [upload=469300]: Done uploading segments... Dec 27 22:35:39.130079990 (535h6m39.001163035s) [upload=469300]: Upload finished. err: <nil> Duration: 15.859773745s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 127.963023ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:36:27 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 128.01447ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X............X.X...X.....X] duration: 30.001793852s trace: https://jaeger.dev.storj.io/trace/7eeb6cfd859a0038 FAIL (2025-12-27 22:36:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q) context deadline exceeded ======LOGS====== Dec 27 22:35:36.307110178 (535h6m36.178193213s) [upload=469300]: Starting upload Dec 27 22:35:36.307180792 (535h6m36.178263837s) [upload=469300]: Done starting upload Dec 27 22:35:36.307286814 (535h6m36.178369859s) [upload=469300]: Uploading segments... Dec 27 22:35:36.307425117 (535h6m36.178508162s) [upload=469300]: Got next segment. Inline: false Dec 27 22:35:36.307476645 (535h6m36.17855968s) [upload=469300]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:35:36.307485442 (535h6m36.178568478s) [upload=469300]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:35:36.550202498 (535h6m36.421285533s) [seg_pos={0 0}, upload=469300]: Begin upload segment... Dec 27 22:35:36.550262552 (535h6m36.421345587s) [seg_pos={0 0}, upload=469300]: Done begin upload segment. Dec 27 22:35:36.550294012 (535h6m36.421377067s) [noise=true , address=145.79.227.7:4293 , piece_id=D2XC5E5SDN3NXRQ2Q3QB7PNULJ744WYEQNZG7RNLEWWLAGB2N3HQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550303389 (535h6m36.421386424s) [noise=true , address=145.79.227.36:4519 , piece_id=GLGX3ROM4Y4QEIENFKNJPLGL47FKXTQLKSBKZTBMULIOTQ27DULQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550352904 (535h6m36.421435939s) [noise=true , address=145.79.227.11:4730 , piece_id=JQM2K2K4OEDGIL4VOOKGMKDWYKT4UADVWTPEL4HXYTLLXRJUIOOQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.550959860 (535h6m36.422042895s) [noise=true , address=145.79.227.12:4765 , piece_id=Y2PZSZUPEXQHXZ7WXO3DGB2JKB2ADX4ZSKCKXF3MNDYTER6XQ6FA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.551337489 (535h6m36.422420534s) [noise=true , address=145.79.227.2:4115 , piece_id=QKYO6ZCJYVBFM6EB6TCCA5P6USGV46MC7CXS7HZSHPYHOIJUY52Q, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.551702815 (535h6m36.42278586s) [noise=true , address=145.79.227.17:4970 , piece_id=UAR77FFRTYRR2VZSMAV4HGXCXDFS6IOEABMWCPX4P5GX3F7L4MHA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.552073340 (535h6m36.423156395s) [noise=true , address=145.79.227.45:4229 , piece_id=PWCS73FOEMLUMAOCYE22YIMJR6EFVR6H44KZ7HBGUUICMQJRMK3A, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.552462371 (535h6m36.423545407s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.553003401 (535h6m36.424086446s) [noise=true , address=77.90.31.13:4627 , piece_id=BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.553460722 (535h6m36.424543767s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554153581 (535h6m36.425236626s) [noise=true , address=145.79.227.4:4185 , piece_id=GLR3OXYB7YYBELFNB6JGXEOITJPC2NE7RO6DSXWGNJ4KD45FHVRA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554610161 (535h6m36.425693196s) [noise=true , address=45.91.116.5:4299 , piece_id=4SRWRWNGVZCBJA52LHJDIGZFX4KQYVKUJHJDBQF4NWHTPSJRH4LA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.554625260 (535h6m36.425708295s) [noise=true , address=45.91.116.13:4153 , piece_id=N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555190296 (535h6m36.426273331s) [noise=true , address=145.79.227.10:4402 , piece_id=AY5DXAWIF6HLGOTDS7HAUC2P6OZK4GBBYRWAKJSE3GCQ4NXPNVSA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555281950 (535h6m36.426364985s) [noise=true , address=145.79.227.52:4446 , piece_id=VGATGH2KIKLQINJPDSNGAXYPMXJMGQVROFP5VCKBI7LYCZIDYLFA, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:36.555890368 (535h6m36.426973414s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Uploading piece... Dec 27 22:35:38.277154486 (535h6m38.148237531s) [upload=469300]: Next returned nil segment Dec 27 22:35:38.277168422 (535h6m38.148251457s) [upload=469300]: Waiting for error group managing segments... Dec 27 22:35:38.555483832 (535h6m38.426566867s) [noise=true , address=145.79.227.11:4730 , piece_id=JQM2K2K4OEDGIL4VOOKGMKDWYKT4UADVWTPEL4HXYTLLXRJUIOOQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.559624757 (535h6m38.430707792s) [noise=true , address=145.79.227.10:4402 , piece_id=AY5DXAWIF6HLGOTDS7HAUC2P6OZK4GBBYRWAKJSE3GCQ4NXPNVSA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.560650140 (535h6m38.431733175s) [noise=true , address=145.79.227.17:4970 , piece_id=UAR77FFRTYRR2VZSMAV4HGXCXDFS6IOEABMWCPX4P5GX3F7L4MHA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.564909921 (535h6m38.435992986s) [noise=true , address=145.79.227.7:4293 , piece_id=D2XC5E5SDN3NXRQ2Q3QB7PNULJ744WYEQNZG7RNLEWWLAGB2N3HQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.572882863 (535h6m38.443965898s) [noise=true , address=145.79.227.45:4229 , piece_id=PWCS73FOEMLUMAOCYE22YIMJR6EFVR6H44KZ7HBGUUICMQJRMK3A, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.573745586 (535h6m38.444828631s) [noise=true , address=145.79.227.4:4185 , piece_id=GLR3OXYB7YYBELFNB6JGXEOITJPC2NE7RO6DSXWGNJ4KD45FHVRA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.574549066 (535h6m38.445632101s) [noise=true , address=145.79.227.36:4519 , piece_id=GLGX3ROM4Y4QEIENFKNJPLGL47FKXTQLKSBKZTBMULIOTQ27DULQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.574760709 (535h6m38.445843744s) [noise=true , address=145.79.227.2:4115 , piece_id=QKYO6ZCJYVBFM6EB6TCCA5P6USGV46MC7CXS7HZSHPYHOIJUY52Q, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.581427843 (535h6m38.452510888s) [noise=true , address=145.79.227.52:4446 , piece_id=VGATGH2KIKLQINJPDSNGAXYPMXJMGQVROFP5VCKBI7LYCZIDYLFA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.757074013 (535h6m38.628157048s) [noise=true , address=145.79.227.12:4765 , piece_id=Y2PZSZUPEXQHXZ7WXO3DGB2JKB2ADX4ZSKCKXF3MNDYTER6XQ6FA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.769644851 (535h6m38.640727886s) [noise=true , address=45.91.116.5:4299 , piece_id=4SRWRWNGVZCBJA52LHJDIGZFX4KQYVKUJHJDBQF4NWHTPSJRH4LA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.814953196 (535h6m38.686036231s) [noise=true , address=45.91.116.13:4153 , piece_id=N6VMHWVMJP2BCZQTIIHOIUPCPGAL4CZNOTJKJZJPSEVSG7LDEO6Q, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.833765998 (535h6m38.704849033s) [noise=true , address=77.90.31.13:4627 , piece_id=BS2NCRXYLRQEVJDFY3XWPKIY6CTMHJIRSEAHVLLJBCSPQLNUDMLQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: <nil> Dec 27 22:35:38.833822626 (535h6m38.704905661s) [seg_pos={0 0}, upload=469300]: Segment reached optimal threshold of 13 pieces. Dec 27 22:35:38.834139189 (535h6m38.705222224s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 22:35:38.834148266 (535h6m38.705231301s) [noise=true , address=77.90.31.18:4805 , piece_id=CBWS7XCCCFTOXUOJKFT6J4J3AZVNXMIZLO4PUO5O4M7KINAP4MYA, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834306488 (535h6m38.705389523s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa): protocol: write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54924->145.79.227.24:4582: use of closed network connection Dec 27 22:35:38.834321235 (535h6m38.70540427s) [noise=true , address=145.79.227.24:4582 , piece_id=46UVQMCECZ3MMG56RZH664TRP7GQCG752DDN5CBEQQYVDXGDCAMQ, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834480158 (535h6m38.705563193s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 27 22:35:38.834486290 (535h6m38.705569335s) [noise=true , address=77.90.31.15:4700 , piece_id=QEDSOXPVDZJI24JCXJY2WEX6EWQAE3QKARFAFM2IZIAP66FKGEGA, seg_pos={0 0}, upload=469300]: Long tail cancellation due to optimal threshold Dec 27 22:35:38.834518411 (535h6m38.705601456s) [seg_pos={0 0}, upload=469300]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:35:38.834595878 (535h6m38.705678913s) [upload=469300]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:35:38.834599966 (535h6m38.705683001s) [upload=469300]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:35:39.130055424 (535h6m39.001138469s) [upload=469300]: Done uploading segments... Dec 27 22:35:39.130079990 (535h6m39.001163035s) [upload=469300]: Upload finished. err: <nil> Duration: 30.001820433s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 152.891281ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:35:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 152.936056ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 141.922865ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:35:32 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 141.944196ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X............X.X...X.....] duration: 6.153251376s trace: https://jaeger.dev.storj.io/trace/325d919d7481d72d FAIL (2025-12-27 22:35:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 22:31:09.524014276 (535h2m9.395097321s) [upload=469238]: Starting upload Dec 27 22:31:09.524089439 (535h2m9.395172484s) [upload=469238]: Done starting upload Dec 27 22:31:09.524320629 (535h2m9.395403675s) [upload=469238]: Uploading segments... Dec 27 22:31:09.524386755 (535h2m9.395469791s) [upload=469238]: Got next segment. Inline: false Dec 27 22:31:09.524422073 (535h2m9.395505118s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:31:09.524427844 (535h2m9.395510889s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:31:09.770678216 (535h2m9.641761261s) [seg_pos={0 0}, upload=469238]: Begin upload segment... Dec 27 22:31:09.770721738 (535h2m9.641804783s) [seg_pos={0 0}, upload=469238]: Done begin upload segment. Dec 27 22:31:09.770766694 (535h2m9.641849729s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.770776463 (535h2m9.641859498s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771493778 (535h2m9.642576813s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771809740 (535h2m9.642892775s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772195284 (535h2m9.643278319s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772509041 (535h2m9.643592076s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772795918 (535h2m9.643878963s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.773093715 (535h2m9.64417675s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774408068 (535h2m9.645491113s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774511916 (535h2m9.645594951s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775538430 (535h2m9.646621465s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775971826 (535h2m9.647054871s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.776648353 (535h2m9.647731389s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.777782564 (535h2m9.648865599s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.779485657 (535h2m9.650568702s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.784191387 (535h2m9.655274432s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:11.504722047 (535h2m11.375805082s) [upload=469238]: Next returned nil segment Dec 27 22:31:11.504745662 (535h2m11.375828697s) [upload=469238]: Waiting for error group managing segments... Dec 27 22:31:11.788222540 (535h2m11.659305585s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.788452679 (535h2m11.659535714s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801119107 (535h2m11.672202152s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801353804 (535h2m11.672436839s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.806234447 (535h2m11.677317482s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.807795811 (535h2m11.678878856s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.824723414 (535h2m11.695806469s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.950312998 (535h2m11.821396033s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.974680398 (535h2m11.845763433s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.020710497 (535h2m11.891793542s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.069247459 (535h2m11.940330504s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.247136981 (535h2m12.118220026s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288435500 (535h2m12.159518535s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288453715 (535h2m12.159536751s) [seg_pos={0 0}, upload=469238]: Segment reached optimal threshold of 13 pieces. Dec 27 22:31:12.288726805 (535h2m12.15980986s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288737305 (535h2m12.15982034s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288780227 (535h2m12.159863262s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288790987 (535h2m12.159874042s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288935592 (535h2m12.160018637s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:33136->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 22:31:12.288942917 (535h2m12.160025952s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288961331 (535h2m12.160044377s) [seg_pos={0 0}, upload=469238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:31:12.289046444 (535h2m12.160129499s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:31:12.289053036 (535h2m12.160136102s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:31:12.580995314 (535h2m12.452078359s) [upload=469238]: Done uploading segments... Dec 27 22:31:12.581027035 (535h2m12.45211007s) [upload=469238]: Upload finished. err: <nil> Duration: 6.153275064s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 168.990196ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:34:21 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 169.018981ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X............X.X...X....] duration: 5.958601602s trace: https://jaeger.dev.storj.io/trace/325d919d7481d72d FAIL (2025-12-27 22:34:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 22:31:09.524014276 (535h2m9.395097321s) [upload=469238]: Starting upload Dec 27 22:31:09.524089439 (535h2m9.395172484s) [upload=469238]: Done starting upload Dec 27 22:31:09.524320629 (535h2m9.395403675s) [upload=469238]: Uploading segments... Dec 27 22:31:09.524386755 (535h2m9.395469791s) [upload=469238]: Got next segment. Inline: false Dec 27 22:31:09.524422073 (535h2m9.395505118s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:31:09.524427844 (535h2m9.395510889s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:31:09.770678216 (535h2m9.641761261s) [seg_pos={0 0}, upload=469238]: Begin upload segment... Dec 27 22:31:09.770721738 (535h2m9.641804783s) [seg_pos={0 0}, upload=469238]: Done begin upload segment. Dec 27 22:31:09.770766694 (535h2m9.641849729s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.770776463 (535h2m9.641859498s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771493778 (535h2m9.642576813s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771809740 (535h2m9.642892775s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772195284 (535h2m9.643278319s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772509041 (535h2m9.643592076s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772795918 (535h2m9.643878963s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.773093715 (535h2m9.64417675s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774408068 (535h2m9.645491113s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774511916 (535h2m9.645594951s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775538430 (535h2m9.646621465s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775971826 (535h2m9.647054871s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.776648353 (535h2m9.647731389s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.777782564 (535h2m9.648865599s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.779485657 (535h2m9.650568702s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.784191387 (535h2m9.655274432s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:11.504722047 (535h2m11.375805082s) [upload=469238]: Next returned nil segment Dec 27 22:31:11.504745662 (535h2m11.375828697s) [upload=469238]: Waiting for error group managing segments... Dec 27 22:31:11.788222540 (535h2m11.659305585s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.788452679 (535h2m11.659535714s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801119107 (535h2m11.672202152s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801353804 (535h2m11.672436839s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.806234447 (535h2m11.677317482s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.807795811 (535h2m11.678878856s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.824723414 (535h2m11.695806469s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.950312998 (535h2m11.821396033s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.974680398 (535h2m11.845763433s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.020710497 (535h2m11.891793542s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.069247459 (535h2m11.940330504s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.247136981 (535h2m12.118220026s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288435500 (535h2m12.159518535s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288453715 (535h2m12.159536751s) [seg_pos={0 0}, upload=469238]: Segment reached optimal threshold of 13 pieces. Dec 27 22:31:12.288726805 (535h2m12.15980986s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288737305 (535h2m12.15982034s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288780227 (535h2m12.159863262s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288790987 (535h2m12.159874042s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288935592 (535h2m12.160018637s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:33136->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 22:31:12.288942917 (535h2m12.160025952s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288961331 (535h2m12.160044377s) [seg_pos={0 0}, upload=469238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:31:12.289046444 (535h2m12.160129499s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:31:12.289053036 (535h2m12.160136102s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:31:12.580995314 (535h2m12.452078359s) [upload=469238]: Done uploading segments... Dec 27 22:31:12.581027035 (535h2m12.45211007s) [upload=469238]: Upload finished. err: <nil> Duration: 5.958620688s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 183.642656ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:33:47 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 183.679776ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X............X.X...X...] duration: 6.044885843s trace: https://jaeger.dev.storj.io/trace/325d919d7481d72d FAIL (2025-12-27 22:33:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 22:31:09.524014276 (535h2m9.395097321s) [upload=469238]: Starting upload Dec 27 22:31:09.524089439 (535h2m9.395172484s) [upload=469238]: Done starting upload Dec 27 22:31:09.524320629 (535h2m9.395403675s) [upload=469238]: Uploading segments... Dec 27 22:31:09.524386755 (535h2m9.395469791s) [upload=469238]: Got next segment. Inline: false Dec 27 22:31:09.524422073 (535h2m9.395505118s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:31:09.524427844 (535h2m9.395510889s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:31:09.770678216 (535h2m9.641761261s) [seg_pos={0 0}, upload=469238]: Begin upload segment... Dec 27 22:31:09.770721738 (535h2m9.641804783s) [seg_pos={0 0}, upload=469238]: Done begin upload segment. Dec 27 22:31:09.770766694 (535h2m9.641849729s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.770776463 (535h2m9.641859498s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771493778 (535h2m9.642576813s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771809740 (535h2m9.642892775s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772195284 (535h2m9.643278319s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772509041 (535h2m9.643592076s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772795918 (535h2m9.643878963s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.773093715 (535h2m9.64417675s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774408068 (535h2m9.645491113s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774511916 (535h2m9.645594951s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775538430 (535h2m9.646621465s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775971826 (535h2m9.647054871s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.776648353 (535h2m9.647731389s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.777782564 (535h2m9.648865599s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.779485657 (535h2m9.650568702s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.784191387 (535h2m9.655274432s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:11.504722047 (535h2m11.375805082s) [upload=469238]: Next returned nil segment Dec 27 22:31:11.504745662 (535h2m11.375828697s) [upload=469238]: Waiting for error group managing segments... Dec 27 22:31:11.788222540 (535h2m11.659305585s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.788452679 (535h2m11.659535714s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801119107 (535h2m11.672202152s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801353804 (535h2m11.672436839s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.806234447 (535h2m11.677317482s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.807795811 (535h2m11.678878856s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.824723414 (535h2m11.695806469s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.950312998 (535h2m11.821396033s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.974680398 (535h2m11.845763433s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.020710497 (535h2m11.891793542s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.069247459 (535h2m11.940330504s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.247136981 (535h2m12.118220026s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288435500 (535h2m12.159518535s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288453715 (535h2m12.159536751s) [seg_pos={0 0}, upload=469238]: Segment reached optimal threshold of 13 pieces. Dec 27 22:31:12.288726805 (535h2m12.15980986s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288737305 (535h2m12.15982034s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288780227 (535h2m12.159863262s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288790987 (535h2m12.159874042s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288935592 (535h2m12.160018637s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:33136->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 22:31:12.288942917 (535h2m12.160025952s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288961331 (535h2m12.160044377s) [seg_pos={0 0}, upload=469238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:31:12.289046444 (535h2m12.160129499s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:31:12.289053036 (535h2m12.160136102s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:31:12.580995314 (535h2m12.452078359s) [upload=469238]: Done uploading segments... Dec 27 22:31:12.581027035 (535h2m12.45211007s) [upload=469238]: Upload finished. err: <nil> Duration: 6.04491553s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 144.708268ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:33:03 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 144.785876ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X............X.X...X..] duration: 10.984648106s trace: https://jaeger.dev.storj.io/trace/325d919d7481d72d FAIL (2025-12-27 22:32:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 22:31:09.524014276 (535h2m9.395097321s) [upload=469238]: Starting upload Dec 27 22:31:09.524089439 (535h2m9.395172484s) [upload=469238]: Done starting upload Dec 27 22:31:09.524320629 (535h2m9.395403675s) [upload=469238]: Uploading segments... Dec 27 22:31:09.524386755 (535h2m9.395469791s) [upload=469238]: Got next segment. Inline: false Dec 27 22:31:09.524422073 (535h2m9.395505118s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:31:09.524427844 (535h2m9.395510889s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:31:09.770678216 (535h2m9.641761261s) [seg_pos={0 0}, upload=469238]: Begin upload segment... Dec 27 22:31:09.770721738 (535h2m9.641804783s) [seg_pos={0 0}, upload=469238]: Done begin upload segment. Dec 27 22:31:09.770766694 (535h2m9.641849729s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.770776463 (535h2m9.641859498s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771493778 (535h2m9.642576813s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771809740 (535h2m9.642892775s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772195284 (535h2m9.643278319s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772509041 (535h2m9.643592076s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772795918 (535h2m9.643878963s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.773093715 (535h2m9.64417675s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774408068 (535h2m9.645491113s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774511916 (535h2m9.645594951s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775538430 (535h2m9.646621465s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775971826 (535h2m9.647054871s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.776648353 (535h2m9.647731389s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.777782564 (535h2m9.648865599s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.779485657 (535h2m9.650568702s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.784191387 (535h2m9.655274432s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:11.504722047 (535h2m11.375805082s) [upload=469238]: Next returned nil segment Dec 27 22:31:11.504745662 (535h2m11.375828697s) [upload=469238]: Waiting for error group managing segments... Dec 27 22:31:11.788222540 (535h2m11.659305585s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.788452679 (535h2m11.659535714s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801119107 (535h2m11.672202152s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801353804 (535h2m11.672436839s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.806234447 (535h2m11.677317482s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.807795811 (535h2m11.678878856s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.824723414 (535h2m11.695806469s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.950312998 (535h2m11.821396033s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.974680398 (535h2m11.845763433s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.020710497 (535h2m11.891793542s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.069247459 (535h2m11.940330504s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.247136981 (535h2m12.118220026s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288435500 (535h2m12.159518535s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288453715 (535h2m12.159536751s) [seg_pos={0 0}, upload=469238]: Segment reached optimal threshold of 13 pieces. Dec 27 22:31:12.288726805 (535h2m12.15980986s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288737305 (535h2m12.15982034s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288780227 (535h2m12.159863262s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288790987 (535h2m12.159874042s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288935592 (535h2m12.160018637s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:33136->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 22:31:12.288942917 (535h2m12.160025952s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288961331 (535h2m12.160044377s) [seg_pos={0 0}, upload=469238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:31:12.289046444 (535h2m12.160129499s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:31:12.289053036 (535h2m12.160136102s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:31:12.580995314 (535h2m12.452078359s) [upload=469238]: Done uploading segments... Dec 27 22:31:12.581027035 (535h2m12.45211007s) [upload=469238]: Upload finished. err: <nil> Duration: 10.984672252s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 171.723481ms trace: https://jaeger.dev.storj.io/trace/65d071dc3d109c6 FAIL (2025-12-27 22:32:34 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 171.765601ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X............X.X...X.] duration: 6.320923735s trace: https://jaeger.dev.storj.io/trace/325d919d7481d72d FAIL (2025-12-27 22:31:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 22:31:09.524014276 (535h2m9.395097321s) [upload=469238]: Starting upload Dec 27 22:31:09.524089439 (535h2m9.395172484s) [upload=469238]: Done starting upload Dec 27 22:31:09.524320629 (535h2m9.395403675s) [upload=469238]: Uploading segments... Dec 27 22:31:09.524386755 (535h2m9.395469791s) [upload=469238]: Got next segment. Inline: false Dec 27 22:31:09.524422073 (535h2m9.395505118s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:31:09.524427844 (535h2m9.395510889s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:31:09.770678216 (535h2m9.641761261s) [seg_pos={0 0}, upload=469238]: Begin upload segment... Dec 27 22:31:09.770721738 (535h2m9.641804783s) [seg_pos={0 0}, upload=469238]: Done begin upload segment. Dec 27 22:31:09.770766694 (535h2m9.641849729s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.770776463 (535h2m9.641859498s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771493778 (535h2m9.642576813s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771809740 (535h2m9.642892775s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772195284 (535h2m9.643278319s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772509041 (535h2m9.643592076s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772795918 (535h2m9.643878963s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.773093715 (535h2m9.64417675s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774408068 (535h2m9.645491113s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774511916 (535h2m9.645594951s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775538430 (535h2m9.646621465s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775971826 (535h2m9.647054871s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.776648353 (535h2m9.647731389s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.777782564 (535h2m9.648865599s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.779485657 (535h2m9.650568702s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.784191387 (535h2m9.655274432s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:11.504722047 (535h2m11.375805082s) [upload=469238]: Next returned nil segment Dec 27 22:31:11.504745662 (535h2m11.375828697s) [upload=469238]: Waiting for error group managing segments... Dec 27 22:31:11.788222540 (535h2m11.659305585s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.788452679 (535h2m11.659535714s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801119107 (535h2m11.672202152s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801353804 (535h2m11.672436839s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.806234447 (535h2m11.677317482s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.807795811 (535h2m11.678878856s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.824723414 (535h2m11.695806469s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.950312998 (535h2m11.821396033s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.974680398 (535h2m11.845763433s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.020710497 (535h2m11.891793542s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.069247459 (535h2m11.940330504s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.247136981 (535h2m12.118220026s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288435500 (535h2m12.159518535s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288453715 (535h2m12.159536751s) [seg_pos={0 0}, upload=469238]: Segment reached optimal threshold of 13 pieces. Dec 27 22:31:12.288726805 (535h2m12.15980986s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288737305 (535h2m12.15982034s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288780227 (535h2m12.159863262s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288790987 (535h2m12.159874042s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288935592 (535h2m12.160018637s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:33136->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 22:31:12.288942917 (535h2m12.160025952s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288961331 (535h2m12.160044377s) [seg_pos={0 0}, upload=469238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:31:12.289046444 (535h2m12.160129499s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:31:12.289053036 (535h2m12.160136102s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:31:12.580995314 (535h2m12.452078359s) [upload=469238]: Done uploading segments... Dec 27 22:31:12.581027035 (535h2m12.45211007s) [upload=469238]: Upload finished. err: <nil> Duration: 6.320952559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................X............X.X...X] duration: 30.001916254s trace: https://jaeger.dev.storj.io/trace/325d919d7481d72d FAIL (2025-12-27 22:31:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 22:31:09.524014276 (535h2m9.395097321s) [upload=469238]: Starting upload Dec 27 22:31:09.524089439 (535h2m9.395172484s) [upload=469238]: Done starting upload Dec 27 22:31:09.524320629 (535h2m9.395403675s) [upload=469238]: Uploading segments... Dec 27 22:31:09.524386755 (535h2m9.395469791s) [upload=469238]: Got next segment. Inline: false Dec 27 22:31:09.524422073 (535h2m9.395505118s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 22:31:09.524427844 (535h2m9.395510889s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 22:31:09.770678216 (535h2m9.641761261s) [seg_pos={0 0}, upload=469238]: Begin upload segment... Dec 27 22:31:09.770721738 (535h2m9.641804783s) [seg_pos={0 0}, upload=469238]: Done begin upload segment. Dec 27 22:31:09.770766694 (535h2m9.641849729s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.770776463 (535h2m9.641859498s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771493778 (535h2m9.642576813s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.771809740 (535h2m9.642892775s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772195284 (535h2m9.643278319s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772509041 (535h2m9.643592076s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.772795918 (535h2m9.643878963s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.773093715 (535h2m9.64417675s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774408068 (535h2m9.645491113s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.774511916 (535h2m9.645594951s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775538430 (535h2m9.646621465s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.775971826 (535h2m9.647054871s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.776648353 (535h2m9.647731389s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.777782564 (535h2m9.648865599s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.779485657 (535h2m9.650568702s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:09.784191387 (535h2m9.655274432s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Uploading piece... Dec 27 22:31:11.504722047 (535h2m11.375805082s) [upload=469238]: Next returned nil segment Dec 27 22:31:11.504745662 (535h2m11.375828697s) [upload=469238]: Waiting for error group managing segments... Dec 27 22:31:11.788222540 (535h2m11.659305585s) [noise=true , address=145.79.227.20:4438 , piece_id=E6TWQJSR3AAEIZ2LFX4VD6AMH5FIRRBKISFPTUBWT2R5LPUWW5UQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.788452679 (535h2m11.659535714s) [noise=true , address=145.79.227.46:4603 , piece_id=77GMYFADPMIMAMRXIXKBWAR52F7PSWIWLJ3V2MCTQTJ4ZXIASFFA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801119107 (535h2m11.672202152s) [noise=true , address=145.79.227.24:4582 , piece_id=QUVUEAOUSQJN6BDMFRTWNDJGJRRUDQP4BZVJRM2UYSG22KXB2NLA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.801353804 (535h2m11.672436839s) [noise=true , address=145.79.227.38:4977 , piece_id=HXWEFDQHNWL2CJMWZOG5ONMAIRZFG2FST7XTQXBTQWNHFBPBIVSA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.806234447 (535h2m11.677317482s) [noise=true , address=145.79.227.3:4151 , piece_id=ELI6FCNKYNIVWGBWUAJWRRDHLQO3NILWWQWQKACM7SCKTCYPA2OQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.807795811 (535h2m11.678878856s) [noise=true , address=145.79.227.36:4518 , piece_id=SMDKCH4W6RYSKVSI3AFHPYCRMZJHAXX7PBIIGUVUCTUAACJHMJDA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.824723414 (535h2m11.695806469s) [noise=true , address=145.79.227.51:4410 , piece_id=HATTH4PIBPNF57ZG4M7OOKVARG5AJGLIROID2CXPEZ5OY7WRQIGA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.950312998 (535h2m11.821396033s) [noise=true , address=77.90.31.19:4844 , piece_id=EZZYF2W6FJV3GZTB6SHFVI6GPFGWW2MBL6S23XC6T7Y6EWF3VDSQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:11.974680398 (535h2m11.845763433s) [noise=true , address=45.91.116.9:4264 , piece_id=UTFGMSEDDNFUSFBORFXMKRZIVZ3WNS7DQYB6D5SDAJJH2W7Y3HKA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.020710497 (535h2m11.891793542s) [noise=true , address=45.91.116.14:4191 , piece_id=SZFK74S6QDKVWBUKCC5JLLVLXSOIWR7WREIHHETAS7RBAJDD7ZRA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.069247459 (535h2m11.940330504s) [noise=true , address=45.91.116.6:4973 , piece_id=YSW3WUNAZTF7STB2SGI3EIXRHFWZ47I2YIXUUPGTHW6DICBK7H4A, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.247136981 (535h2m12.118220026s) [noise=true , address=45.91.116.4:4939 , piece_id=FPM4VVSIPZDBQNOQ6K7GZGMGVDUVNUUXB7IAF24UPR3TRQ4LGGKQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288435500 (535h2m12.159518535s) [noise=true , address=77.90.31.18:4808 , piece_id=KSKW5YANJQK73BMH5GIK7GN5AV3UU3OQKKIKEPLQKQMWRCCEI7RA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: <nil> Dec 27 22:31:12.288453715 (535h2m12.159536751s) [seg_pos={0 0}, upload=469238]: Segment reached optimal threshold of 13 pieces. Dec 27 22:31:12.288726805 (535h2m12.15980986s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288737305 (535h2m12.15982034s) [noise=true , address=145.79.227.40:4050 , piece_id=ZLIYR4YO5LBMU3IYHVNYYBNPXUGABZPA7TRMOT3SU3BKHSB42BNA, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288780227 (535h2m12.159863262s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 22:31:12.288790987 (535h2m12.159874042s) [noise=true , address=45.91.116.11:4120 , piece_id=2OR6KDUNI5RLB37WTFJJISPRYCOKS7ZZECROUEFB5HB6ZOBBKMMQ, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288935592 (535h2m12.160018637s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:33136->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 22:31:12.288942917 (535h2m12.160025952s) [noise=true , address=45.91.116.10:4081 , piece_id=DM3IK24JDEOYAZNJPEJ3AA4X4FHJE776YF2EOI7KK5YDFYNIUO4Q, seg_pos={0 0}, upload=469238]: Long tail cancellation due to optimal threshold Dec 27 22:31:12.288961331 (535h2m12.160044377s) [seg_pos={0 0}, upload=469238]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 22:31:12.289046444 (535h2m12.160129499s) [upload=469238]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 22:31:12.289053036 (535h2m12.160136102s) [upload=469238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 22:31:12.580995314 (535h2m12.452078359s) [upload=469238]: Done uploading segments... Dec 27 22:31:12.581027035 (535h2m12.45211007s) [upload=469238]: Upload finished. err: <nil> Duration: 30.001939929s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)