Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 26.705226549s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:46:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 26.705254462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X...........X..................X.......X....] duration: 5.927284218s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:45:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 5.927326538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 15.387015112s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:45:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 15.38704532s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X...........X..................X.......X...] duration: 15.882161917s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:45:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 15.882192206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 33.701487384s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:44:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 33.701521469s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X...........X..................X.......X..] duration: 13.497161673s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:44:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 13.497188145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 14.874941604s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:44:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 14.874960389s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X...........X..................X.......X.] duration: 6.16747382s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:43:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 6.167502605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 10.007722622s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:43:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 10.007792014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X...........X..................X.......X] duration: 30.003100549s trace: https://jaeger.dev.storj.io/trace/27073096d2c66e66 FAIL (2025-12-28 20:43:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ) context deadline exceeded ======LOGS====== g segments... Dec 28 20:42:45.662246218 (557h13m45.533329253s) [upload=488520]: Got next segment. Inline: false Dec 28 20:42:45.662270836 (557h13m45.533353881s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:42:45.662274963 (557h13m45.533357998s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:42:45.907695937 (557h13m45.778778992s) [seg_pos={0 0}, upload=488520]: Begin upload segment... Dec 28 20:42:45.907765980 (557h13m45.778849015s) [seg_pos={0 0}, upload=488520]: Done begin upload segment. Dec 28 20:42:45.907818851 (557h13m45.778901886s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.907835683 (557h13m45.778918728s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908356295 (557h13m45.77943934s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.908784009 (557h13m45.779867054s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909197486 (557h13m45.780280531s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909211162 (557h13m45.780294207s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909627475 (557h13m45.78071052s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909651611 (557h13m45.780734656s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.909972622 (557h13m45.781055657s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910169056 (557h13m45.781252101s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910608624 (557h13m45.781691659s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.910880450 (557h13m45.781963485s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911138993 (557h13m45.782222038s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911273580 (557h13m45.782356615s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911547812 (557h13m45.782630857s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:45.911971338 (557h13m45.783054383s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Uploading piece... Dec 28 20:42:47.615828034 (557h13m47.486911079s) [upload=488520]: Next returned nil segment Dec 28 20:42:47.615867419 (557h13m47.486950454s) [upload=488520]: Waiting for error group managing segments... Dec 28 20:42:47.902889789 (557h13m47.773972834s) [noise=true , address=145.79.227.37:4565 , piece_id=6GSLWIFYPMOS3DIYSJHBIGD6OMBOFNUMLLST7AUXICEDCZBDJ5LA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.911211395 (557h13m47.78229443s) [noise=true , address=145.79.227.15:4875 , piece_id=WP727FDSSU2NI5T2L7VYLJOSMJ4VOQN5PYFJIZRVWZ7JX5N6MKWA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915667801 (557h13m47.786750836s) [noise=true , address=145.79.227.39:4013 , piece_id=JVIOUEJBGKXM33NHZEL4O3TA3UAHA2LCV27AYFTCSEKF7I4OJZXA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.915898529 (557h13m47.786981564s) [noise=true , address=145.79.227.16:4911 , piece_id=VADBIEKNEJKZ6E22T2JM6O7QLHWHDCCFKZAUC3YKURVQBBAIDWKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.916613942 (557h13m47.787696987s) [noise=true , address=145.79.227.45:4231 , piece_id=RA5ZQVPR42YPOQG6S6SG6FFWRIM4STOF2L6IUWSD5YIBDPNVU3SQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.919951998 (557h13m47.791035033s) [noise=true , address=145.79.227.19:4077 , piece_id=74G3RRZP4KLQIU6SUZ3OPH7JYMFYLWVIO3YDO7MVMCT7JAOXT7IQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:47.926441575 (557h13m47.79752462s) [noise=true , address=145.79.227.46:4602 , piece_id=5NNVLQOPKQSKXMJPA2VM5HFE5RFV5BBQLEFUUI23QVHVGNBQSV6Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.144918112 (557h13m48.016001157s) [noise=true , address=45.91.116.5:4300 , piece_id=CRR2UNPJ5AL263FKPAG553XNG52M4OLO4KAGGSCIQPASTGJFLC5A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.149275801 (557h13m48.020358836s) [noise=true , address=77.90.31.21:4878 , piece_id=HMF5EGQKGM2YEBY3OGIOXMVDMCUSGZHFMISWLUFETMPX6TXQGNUQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.205776737 (557h13m48.076859782s) [noise=true , address=77.90.31.14:4661 , piece_id=HLBOPWGXKQBLUCMFUZVHGOR3GRF5PQXTDEQZYXR7DOINQQWOKFPQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.219929196 (557h13m48.091012241s) [noise=true , address=77.90.31.15:4697 , piece_id=CNBN2IQKKV3BN5SPAFLZ6ZB4NLL6SVVQDISYAQPDYWWVNKRSU45A, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.227276968 (557h13m48.098360013s) [noise=true , address=45.91.116.12:4227 , piece_id=VE6AZM6VXOSYVYC2Q3TWSR74QOV43BHZ2CQQO6RSM34RVAD5MKJQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235297289 (557h13m48.106380334s) [noise=true , address=45.91.116.4:4937 , piece_id=K3WLF64M4S24U4AEPEFBJKIJXQ47AVTM4D3GARCOFHCXSGLILOKQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: <nil> Dec 28 20:42:48.235330643 (557h13m48.106413678s) [seg_pos={0 0}, upload=488520]: Segment reached optimal threshold of 13 pieces. Dec 28 20:42:48.235563236 (557h13m48.106646271s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59310->77.90.31.12:4590: use of closed network connection Dec 28 20:42:48.235571932 (557h13m48.106654978s) [noise=true , address=77.90.31.12:4590 , piece_id=V7WS7Y6UCOCREAIBLK5V5CAF4UHF6LQ36V5BOPHSZJZBC5MRSM4Q, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235663838 (557h13m48.106746873s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 20:42:48.235688946 (557h13m48.106771991s) [noise=true , address=145.79.227.35:4482 , piece_id=D4TRJTUIJG763XYZ7EGTICLOMJZWC4XCCAJWQ63FDXBK3VUHQ6BQ, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235931368 (557h13m48.107014403s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36760->77.90.31.19:4842: use of closed network connection Dec 28 20:42:48.235940565 (557h13m48.10702361s) [noise=true , address=77.90.31.19:4842 , piece_id=DJIYFXSNF3LPOCFXPBBAPHD5MV4X5RKFCC66AEQEVTEA6NZR2MZA, seg_pos={0 0}, upload=488520]: Long tail cancellation due to optimal threshold Dec 28 20:42:48.235960813 (557h13m48.107043848s) [seg_pos={0 0}, upload=488520]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:42:48.236068709 (557h13m48.107151754s) [upload=488520]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:42:48.236076794 (557h13m48.107159829s) [upload=488520]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:42:48.536044414 (557h13m48.407127459s) [upload=488520]: Done uploading segments... Dec 28 20:42:48.536067258 (557h13m48.407150293s) [upload=488520]: Upload finished. err: <nil> Duration: 30.003121779s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 33.785015085s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:43:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 33.78504433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X...........X..................X.......] duration: 6.17911129s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:42:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 6.179497444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X...........X..................X......] duration: 24.396574929s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:41:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 24.396594566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 25.359651363s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:41:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 25.359679516s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 36.926128315s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:41:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 36.926158573s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X...........X..................X.....] duration: 7.059598399s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:40:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 7.059646561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X...........X..................X....] duration: 20.172161812s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:40:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 20.172191941s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 25.331743784s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:39:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 25.331771536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X...........X..................X...] duration: 6.416913838s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:39:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 6.416931712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X...........X..................X..] duration: 28.074753124s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:39:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 28.074792137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 41.02160755s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:39:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 41.021637247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..........................X...........X..................X.] duration: 5.83937331s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:38:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 5.839402254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..........................X...........X..................X] duration: 30.002949498s trace: https://jaeger.dev.storj.io/trace/60ddd33e73f0de61 FAIL (2025-12-28 20:37:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 065810223s) [upload=488441]: Uploading segments... Dec 28 20:37:27.194835273 (557h8m27.065918308s) [upload=488441]: Got next segment. Inline: false Dec 28 20:37:27.194877263 (557h8m27.065960308s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:37:27.194884157 (557h8m27.065967212s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:37:27.450350479 (557h8m27.321433524s) [seg_pos={0 0}, upload=488441]: Begin upload segment... Dec 28 20:37:27.450441583 (557h8m27.321524628s) [seg_pos={0 0}, upload=488441]: Done begin upload segment. Dec 28 20:37:27.450581259 (557h8m27.321664304s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.450619542 (557h8m27.321702597s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451283105 (557h8m27.32236617s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.451709617 (557h8m27.322792662s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452173731 (557h8m27.323256766s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.452596105 (557h8m27.32367914s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453086699 (557h8m27.324169734s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.453622139 (557h8m27.324705184s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.455648808 (557h8m27.326731853s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.456606212 (557h8m27.327689247s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.457955700 (557h8m27.329038735s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.459219468 (557h8m27.330302513s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.460412208 (557h8m27.331495254s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.461658171 (557h8m27.332741206s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.462997892 (557h8m27.334080927s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:27.465119382 (557h8m27.336202427s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Uploading piece... Dec 28 20:37:29.157770565 (557h8m29.02885361s) [upload=488441]: Next returned nil segment Dec 28 20:37:29.157787036 (557h8m29.028870091s) [upload=488441]: Waiting for error group managing segments... Dec 28 20:37:29.455180586 (557h8m29.326263631s) [noise=true , address=145.79.227.26:4654 , piece_id=XZZEQ7Y3SLK6WJJ22HH5RUTABJIYTONKTITYKENJXREOXSNSDHIA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.457221773 (557h8m29.328304808s) [noise=true , address=145.79.227.6:4257 , piece_id=DSINVYJEDHKBN7OGTQZM34CT6MBFUZMALJZQFOGTS5CLTSZLD4CA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.458700027 (557h8m29.329783062s) [noise=true , address=145.79.227.11:4731 , piece_id=R3RGSPD3B7K72ZH4AWXSR6K6D5KYZYLXIOM6QB346S4XZOGKR24A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.459240897 (557h8m29.330323933s) [noise=true , address=145.79.227.51:4410 , piece_id=HGHWLMHV7LMCQGNQGSFRXUGJUVOIPYBPT3XATCP4EBZ2N24YW5LA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.462155848 (557h8m29.333238883s) [noise=true , address=145.79.227.15:4873 , piece_id=2MYCI55SJDFGKGRPUC2BLKB4XXLTFHCNH65JEV6KADXDBJUNBH3Q, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.465572805 (557h8m29.33665585s) [noise=true , address=145.79.227.32:4835 , piece_id=NIDZUDIX6K564WEVU7QDG2DH76GODGMLOJVUVLH6GS7GMUV45D7A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.470903064 (557h8m29.341986109s) [noise=true , address=145.79.227.2:4114 , piece_id=MTU5VYRZVWYYP44THTSGDURQ3WHCCNH2C3XJJYQLUD7JBXUU5NZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.485295701 (557h8m29.356378736s) [noise=true , address=145.79.227.37:4565 , piece_id=3EJIDJFRGBG6272N24VFWNAXKUW7DHHM563DGJTMRURHA27CEREQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.497253901 (557h8m29.368336936s) [noise=true , address=145.79.227.14:4840 , piece_id=VVOFKZVGGDZG6E75IVAUFUGSFNAMN5STZIPSURJ64JPGOWY6BJZQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.641989392 (557h8m29.513072437s) [noise=true , address=77.90.31.16:4736 , piece_id=6NKLMJS4Z4DRFNCOYKO36LAI4EUWFB7LVXEYFDUYUMKSX6R235TQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.701668272 (557h8m29.572751307s) [noise=true , address=45.91.116.5:4299 , piece_id=6UHFIECEOY2EGUKVFY2T5STZWL4DXXFP3NY4WVZKEEIREZ7ZVX2A, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.702897923 (557h8m29.573980958s) [noise=true , address=45.91.116.6:4975 , piece_id=DCEIFZV6754QAYC5N2SXZ5DIYC7NR6TMHCXD66T4A5JRL7E4LPHA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762609574 (557h8m29.633692619s) [noise=true , address=45.91.116.7:4012 , piece_id=C4J7O54YJEGULQT2ZWHPSWTZUU2LU2MN2ND2HSTJI2E6F7OHJSRQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: <nil> Dec 28 20:37:29.762632388 (557h8m29.633715433s) [seg_pos={0 0}, upload=488441]: Segment reached optimal threshold of 13 pieces. Dec 28 20:37:29.762867626 (557h8m29.633950671s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44926->145.79.227.5:4222: use of closed network connection Dec 28 20:37:29.762890609 (557h8m29.633973644s) [noise=true , address=145.79.227.5:4222 , piece_id=IBMJHO2KIDNN5AIIEGX5G5QVDJXXGLOTAMOQC4LZDUBBGY5E6GLQ, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.762944091 (557h8m29.634027126s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 20:37:29.762949461 (557h8m29.634032496s) [noise=true , address=45.91.116.4:4939 , piece_id=XQHY6GBPJ6XEOFL2FTOFNBRLD3GF6VLGUBWY2D7TUOPQSFERNQTA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763074540 (557h8m29.634157585s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54746->145.79.227.27:4690: use of closed network connection Dec 28 20:37:29.763081002 (557h8m29.634164037s) [noise=true , address=145.79.227.27:4690 , piece_id=SZGKSH5YEIQBDSIMVFXRL2X6PXVXGYZ4DOFXWWE6UDQ4CEFTQHJA, seg_pos={0 0}, upload=488441]: Long tail cancellation due to optimal threshold Dec 28 20:37:29.763098996 (557h8m29.634182031s) [seg_pos={0 0}, upload=488441]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:37:29.763231739 (557h8m29.634314774s) [upload=488441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:37:29.763236939 (557h8m29.634319984s) [upload=488441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:37:30.055521760 (557h8m29.926604805s) [upload=488441]: Done uploading segments... Dec 28 20:37:30.055544604 (557h8m29.926627639s) [upload=488441]: Upload finished. err: <nil> Duration: 30.002976549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 17.679037293s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:37:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 17.679063934s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..........................X...........X..................] duration: 7.594979472s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:37:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 7.595006132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 21.010764004s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:36:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 21.010791677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..........................X...........X.................] duration: 28.689432041s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:36:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 28.689449756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 15.531225894s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:35:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 15.531246443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..........................X...........X................] duration: 6.62774145s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:35:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.627770555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 35.145263596s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:34:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 35.145296991s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........................X...........X...............] duration: 6.067094558s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:34:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.067128553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 23.042779417s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:34:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 23.042795808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..........................X...........X..............] duration: 29.366848352s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:34:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 29.366869703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 41.922543615s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:33:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 41.922569634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.002882386s trace: https://jaeger.dev.storj.io/trace/499b96822b8dda80 FAIL (2025-12-28 20:33:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12S7WBBQ1Nq8jbeGrZiRKTk23CdJC31hPW2vU1aaM5SsHeVyeRH, Piece ID: V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q) context deadline exceeded; piecestore: (Node ID: 1kjCEPtjvcJ3D7BBKRSRirPybBwkRZ1dJwyAQCadzCvaLYvWhz, Piece ID: 4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ) context deadline exceeded; piecestore: (Node ID: 1F7WDweCEEWG1kxAAQpJPcigEHsrjuqpuGhGTzNPPYTsXEX43c, Piece ID: LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q) context deadline exceeded ======LOGS====== ue , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496428825 (557h3m1.36751187s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496861509 (557h3m1.367944544s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.496899933 (557h3m1.367982978s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497229089 (557h3m1.368312134s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497326194 (557h3m1.368409239s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497664128 (557h3m1.368747173s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.497731405 (557h3m1.36881445s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498205939 (557h3m1.369288984s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498424295 (557h3m1.36950734s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.498685422 (557h3m1.369768457s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499066348 (557h3m1.370149393s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499185725 (557h3m1.37026877s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499468122 (557h3m1.370551157s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.499901047 (557h3m1.370984102s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:01.500546175 (557h3m1.37162921s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Uploading piece... Dec 28 20:32:02.979572381 (557h3m2.850655436s) [upload=488362]: Next returned nil segment Dec 28 20:32:02.979609411 (557h3m2.850692456s) [upload=488362]: Waiting for error group managing segments... Dec 28 20:32:03.416896733 (557h3m3.287979778s) [noise=true , address=145.79.227.23:4580 , piece_id=OHYI2SFM6TKJE4LHVNGO2PVGZDNFIQDAIQGKG4FOEHTTASBWA5ZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.552263655 (557h3m3.4233467s) [noise=true , address=77.90.31.17:4784 , piece_id=JV2ROD6MN65B6LZKOL3S7ATJWXAY6FBQSZWITA6IHAXELY7R42KA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.573564856 (557h3m3.444647902s) [noise=true , address=77.90.31.12:4602 , piece_id=2YPYKKTQ6ID6VTI4AUZQEG2SY7JYPRSAPU4JLF7DWSOSBBO74LWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.598560923 (557h3m3.469643968s) [noise=true , address=77.90.31.19:4867 , piece_id=LDAWD2ZFG3BZATDRY7MMNWYIX6TOHQ2OIUPCAWAYN266GNUUKZ5A, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.626925925 (557h3m3.49800897s) [noise=true , address=77.90.31.16:4745 , piece_id=CZBFU33BHMP2BEZIDJKW3J526537HR6HP7OCT2KMAQVY7LGZLPZQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.709061947 (557h3m3.580144992s) [noise=true , address=77.90.31.14:4688 , piece_id=4OULHJ6Z3QJ3BOPDY74TJSKVJG23TOEUCJCUEGY24I5C3MUTAVLQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:03.788879373 (557h3m3.659962418s) [noise=true , address=77.90.31.21:4895 , piece_id=V4P3KCUDLI543TSROBYUMVTQKKSQDDVSKSJSXKEZTQAHTXUJ2G4Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.217616523 (557h3m4.088699569s) [noise=true , address=77.90.31.19:4858 , piece_id=LIYXCPAZOUPKXLKRH3EOWREXUY2LAJXSRZFQILM2QIU7HTW7SY2Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.220337625 (557h3m4.09142067s) [noise=true , address=77.90.31.11:4563 , piece_id=DIFJPHSKHROTLKCHP6HQBWT7XGE77HVRZTH2354MV2UEWZLHVGCQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.303958744 (557h3m4.175041789s) [noise=true , address=145.79.227.44:4213 , piece_id=UEAQDBBCP7DIG6ZD3EEM2CKAL4W5QKW23LWFUAMSVUW5ZS2PJDRQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.312794398 (557h3m4.183877444s) [noise=true , address=145.79.227.34:4918 , piece_id=SPDT2KI2774UJMQIUSVTHXCDF7PD2FBG64BXUYW67UMLPE6H7SBQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.349308798 (557h3m4.220391833s) [noise=true , address=145.79.227.40:4058 , piece_id=EQ4LA7T2BCGWY76K3NJI4X77OQJFJED6H4VNGXDQMILU6IVQWOMQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.371518379 (557h3m4.242601424s) [noise=true , address=145.79.227.43:4171 , piece_id=YF5DRDPGRRVYFL5SWMH2PKTSV3C72BIWKUDSAIZTXOIVOTYKRMNA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376232316 (557h3m4.247315361s) [noise=true , address=145.79.227.30:4768 , piece_id=KK4LEV6RG3IGLYXID5MJF3KMNSJ7WS5JS45PNOOJ26LXLYAUCWOA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: <nil> Dec 28 20:32:04.376257844 (557h3m4.247340879s) [seg_pos={0 0}, upload=488362]: Segment reached optimal threshold of 14 pieces. Dec 28 20:32:04.376575799 (557h3m4.247658834s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.376585127 (557h3m4.247668163s) [noise=true , address=145.79.227.33:4904 , piece_id=QCMN4OSBWIWS4JRLIGQIGDSHZFUF2JWEPVXQ4FQ4G3LG5FBCBAWQ, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376764359 (557h3m4.247847404s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GhDrKMcDEg2RxamqKtiHphpgNdvtZYdHXrjzGsNJFjBFYeEen): protocol: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38838->145.79.227.19:4105: use of closed network connection Dec 28 20:32:04.376772865 (557h3m4.2478559s) [noise=true , address=145.79.227.19:4105 , piece_id=RY35FSM6NTWPNF3JE43FZRSNXXYZPUN2B6LRNNUOG6QQR6XSU36Q, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.376926658 (557h3m4.248009703s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Dec 28 20:32:04.377165332 (557h3m4.248248377s) [noise=true , address=145.79.227.35:4496 , piece_id=JH4XQBT7XYEJXWBGPPF6SJWCPLV77GBOLGOA53C3WUQMWHZXVZQA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377212682 (557h3m4.248295848s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW): protocol: read tcp 188.166.225.170:55416->45.91.116.5:4330: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 20:32:04.377289859 (557h3m4.248372924s) [noise=true , address=45.91.116.5:4330 , piece_id=EVHLSVVR5JGY43TLSKVEQLQ6KDBEZENVAFL6YT7ZQ77Y7X2TXIKA, seg_pos={0 0}, upload=488362]: Long tail cancellation due to optimal threshold Dec 28 20:32:04.377387736 (557h3m4.248470781s) [seg_pos={0 0}, upload=488362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 20:32:04.377599729 (557h3m4.248682774s) [upload=488362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:32:04.377613535 (557h3m4.24869662s) [upload=488362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:32:04.675090855 (557h3m4.5461739s) [upload=488362]: Done uploading segments... Dec 28 20:32:04.675121233 (557h3m4.546204268s) [upload=488362]: Upload finished. err: <nil> Duration: 1m0.002894038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........................X...........X.............] duration: 10.279907573s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:32:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 10.279929916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........................X...........X............] duration: 6.302933279s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:32:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.303043088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X..........................X...........X...........] duration: 7.456667189s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:31:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 7.456691265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X..........................X...........X..........] duration: 5.848361324s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:30:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 5.848388977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X..........................X...........X.........] duration: 7.621271797s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:29:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 7.621305661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..........................X...........X........] duration: 6.123378173s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:29:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.123410645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..........................X...........X.......] duration: 5.998546724s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:28:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 5.998573995s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..........................X...........X......] duration: 11.086692143s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:27:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 11.086715566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..............X..........................X...........X.....] duration: 6.08714837s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:27:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.087177294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX..............X..........................X...........X....] duration: 6.021869935s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:26:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.021888019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX..............X..........................X...........X...] duration: 6.042530476s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:25:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 6.042598707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX..............X..........................X...........X..] duration: 23.17607787s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:24:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 23.176109361s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 21.965806753s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:24:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 21.965834125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX..............X..........................X...........X.] duration: 5.8736106s trace: https://jaeger.dev.storj.io/trace/3425d363be458a81 FAIL (2025-12-28 20:24:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA) context deadline exceeded ======LOGS====== Dec 28 20:22:57.518307420 (556h53m57.389390455s) [upload=488235]: Starting upload Dec 28 20:22:57.518369106 (556h53m57.389452182s) [upload=488235]: Done starting upload Dec 28 20:22:57.518427107 (556h53m57.389510152s) [upload=488235]: Uploading segments... Dec 28 20:22:57.518663427 (556h53m57.389746473s) [upload=488235]: Got next segment. Inline: false Dec 28 20:22:57.518709805 (556h53m57.38979284s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 20:22:57.518716318 (556h53m57.389799353s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 20:22:57.767659750 (556h53m57.638742815s) [seg_pos={0 0}, upload=488235]: Begin upload segment... Dec 28 20:22:57.767735904 (556h53m57.638818949s) [seg_pos={0 0}, upload=488235]: Done begin upload segment. Dec 28 20:22:57.767792252 (556h53m57.638875287s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.767877795 (556h53m57.63896085s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768512193 (556h53m57.639595238s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.768539074 (556h53m57.639622119s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769129117 (556h53m57.640212152s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.769521375 (556h53m57.64060441s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.770627170 (556h53m57.641710205s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.771369815 (556h53m57.64245285s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.772492252 (556h53m57.643575297s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.773128333 (556h53m57.644211368s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.775274971 (556h53m57.646358006s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.777926579 (556h53m57.649009615s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.780387376 (556h53m57.651470421s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.781667764 (556h53m57.652750809s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.783365686 (556h53m57.654448731s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:57.784872727 (556h53m57.655955762s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Uploading piece... Dec 28 20:22:59.482505035 (556h53m59.35358808s) [upload=488235]: Next returned nil segment Dec 28 20:22:59.482533810 (556h53m59.353616845s) [upload=488235]: Waiting for error group managing segments... Dec 28 20:22:59.769352743 (556h53m59.640435799s) [noise=true , address=145.79.227.18:4007 , piece_id=NB6ADK5TXELRFCYMKDF6C3QZRZQUL5XUPVQ7AVVYP32UY2GZA6SA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.782950737 (556h53m59.654033782s) [noise=true , address=145.79.227.51:4409 , piece_id=C342XC375VQVI23V5JESCTOES4AF66QWHIETEOBVV3KGRLIP2X5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.789678698 (556h53m59.660761733s) [noise=true , address=145.79.227.12:4765 , piece_id=IESWSXDQFYKKHDOEHSYQSVMTBYLAJWNQ5TPWGNQPMLHO23CXYXTQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.802190252 (556h53m59.673273287s) [noise=true , address=145.79.227.42:4123 , piece_id=TN4ZO3UUKIFRAP44SANXU6HKESFMIE45S5XTWAIKHJW5SA6C2KFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.804531330 (556h53m59.675614375s) [noise=true , address=145.79.227.50:4375 , piece_id=DMDQHEH7XJT334MFCFAPNCGLSJ4UWE3U74MTDFROMTM4BDAI4IHA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.810012928 (556h53m59.681095963s) [noise=true , address=145.79.227.21:4473 , piece_id=OHS3FHPTXD3EKQNBJLLGKENQIJPKPGEFTVIRVQYXO3PZQEVQILGQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812050608 (556h53m59.683133643s) [noise=true , address=145.79.227.36:4519 , piece_id=YIPTLMBP7UV4GLPNA74M4343U4JEE7433ANXUHKYIJLZ36T25TXA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.812993363 (556h53m59.684076398s) [noise=true , address=145.79.227.41:4086 , piece_id=YRHLQZ64IEO7C26HZTCDY2T2PRYUFP6CSN2EOBJS7XAE4I7MXLHQ, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.946324830 (556h53m59.817407865s) [noise=true , address=77.90.31.13:4627 , piece_id=XB2K7RONOVEBPA2XGY36WIE57MWUUBKVAHEGXHAZWNDRL22WHH5Q, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:22:59.965566208 (556h53m59.836649243s) [noise=true , address=77.90.31.17:4770 , piece_id=MH6LXF5JVTVI3U3N7W73U4ULOARVA6G2CRNUQLQ3V2UE6HUDGCUA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.024382405 (556h53m59.89546544s) [noise=true , address=77.90.31.21:4880 , piece_id=V4PF2AQHKOK3GCF6B4UDNRJW7WYQ5QVOC66HUJMOPOOLPZAQIM4A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.056351529 (556h53m59.927434574s) [noise=true , address=45.91.116.5:4297 , piece_id=FNX3RGGDGJZL5DOZRL2ZF2K2JHRAHWAWHIN6NQ4ONUKPVC6J4ECA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067006559 (556h53m59.938089594s) [noise=true , address=45.91.116.13:4155 , piece_id=HHINRA5KN56ZF2KW7IEG4BUGXM6UKMNUM3TM4Y3HFADVWASFEHFA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: <nil> Dec 28 20:23:00.067025144 (556h53m59.938108179s) [seg_pos={0 0}, upload=488235]: Segment reached optimal threshold of 13 pieces. Dec 28 20:23:00.067214014 (556h53m59.938297059s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60580->77.90.31.11:4556: use of closed network connection Dec 28 20:23:00.067225536 (556h53m59.938308571s) [noise=true , address=77.90.31.11:4556 , piece_id=QRRGWDO7EHFVVQ6X4BTVY7HQRRDF5UCHKCFRLO2FOOYLTEDNJR2A, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067306319 (556h53m59.938389364s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067314354 (556h53m59.938397389s) [noise=true , address=77.90.31.15:4697 , piece_id=25QRUIDDPIZ7CSGKDAXLMZIWJHE6HHITH6FU4H2GA2GDBI4PR5KA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067582064 (556h53m59.938665119s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 28 20:23:00.067589408 (556h53m59.938672453s) [noise=true , address=145.79.227.40:4051 , piece_id=PZ3547GPNB3TRB24RBQVWLHLOC5JQVFUMG4ZZJDA3BBESZN3XFYA, seg_pos={0 0}, upload=488235]: Long tail cancellation due to optimal threshold Dec 28 20:23:00.067605620 (556h53m59.938688655s) [seg_pos={0 0}, upload=488235]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 20:23:00.067681614 (556h53m59.938764659s) [upload=488235]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 20:23:00.067686654 (556h53m59.938769699s) [upload=488235]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 20:23:00.354014782 (556h54m0.225097827s) [upload=488235]: Done uploading segments... Dec 28 20:23:00.354034600 (556h54m0.225117635s) [upload=488235]: Upload finished. err: <nil> Duration: 5.873641418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 19.238494732s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:23:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 19.238527204s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)