Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 30.001799301s FAIL (2025-10-26 17:18:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 30.001825531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.84225334s FAIL (2025-10-26 17:17:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.842288297s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.908819561s FAIL (2025-10-26 17:16:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.90883977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.990247579s FAIL (2025-10-26 17:16:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.990281163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.549538286s FAIL (2025-10-26 17:15:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.549578522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 8.214527534s FAIL (2025-10-26 17:14:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 8.214558003s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.04514364s FAIL (2025-10-26 17:13:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.045172394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.004520447s FAIL (2025-10-26 17:13:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.004557207s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.847481799s FAIL (2025-10-26 17:12:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.847517687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.075592491s FAIL (2025-10-26 17:11:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.075615103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.948848208s FAIL (2025-10-26 17:10:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.948879026s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.883001722s FAIL (2025-10-26 17:09:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.883035176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.220389423s FAIL (2025-10-26 17:08:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.22042542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.930595022s FAIL (2025-10-26 17:07:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.93063499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.87820772s FAIL (2025-10-26 17:06:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.878263175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.90646433s FAIL (2025-10-26 17:05:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.906498055s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.142851632s FAIL (2025-10-26 17:05:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.142885616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.082327934s FAIL (2025-10-26 17:04:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.082371526s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.074131676s FAIL (2025-10-26 17:04:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.074166121s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.171858858s FAIL (2025-10-26 17:03:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.171911558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.066240089s FAIL (2025-10-26 17:01:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.066271238s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.918872313s FAIL (2025-10-26 17:01:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.918918992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.007666286s FAIL (2025-10-26 17:00:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.007709827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.846584304s FAIL (2025-10-26 16:59:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.846614981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.043455612s FAIL (2025-10-26 16:59:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.043491862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.159816136s FAIL (2025-10-26 16:58:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.159847004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.17482761s FAIL (2025-10-26 16:56:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.17486485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.363706749s FAIL (2025-10-26 16:56:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.363736647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.939506413s FAIL (2025-10-26 16:55:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.939540088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.062538383s FAIL (2025-10-26 16:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.062570594s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.820080636s FAIL (2025-10-26 16:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.820111695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.187072482s FAIL (2025-10-26 16:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.187105303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.046491022s FAIL (2025-10-26 16:53:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.046525267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.986028429s FAIL (2025-10-26 16:52:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.986063807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.904320653s FAIL (2025-10-26 16:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.904357222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.99629071s FAIL (2025-10-26 16:51:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.996325136s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.112830195s FAIL (2025-10-26 16:50:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.112857378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 8.627961234s FAIL (2025-10-26 16:49:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 8.627989909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.944279358s FAIL (2025-10-26 16:48:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.944313383s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.146287843s FAIL (2025-10-26 16:48:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.146320446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.887680876s FAIL (2025-10-26 16:47:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.887708389s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.04984134s FAIL (2025-10-26 16:46:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.049873349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.945142114s FAIL (2025-10-26 16:45:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 5.945174096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.100032715s FAIL (2025-10-26 16:45:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.100083542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.025349346s FAIL (2025-10-26 16:44:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 6.025381937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002909817s FAIL (2025-10-26 16:44:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: 4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q) context deadline exceeded ======LOGS====== Oct 26 16:43:34.695534669 (795h12m20.157627142s) [upload=699478]: Starting upload Oct 26 16:43:34.695632055 (795h12m20.157724538s) [upload=699478]: Done starting upload Oct 26 16:43:34.695751111 (795h12m20.157843584s) [upload=699478]: Uploading segments... Oct 26 16:43:34.695837156 (795h12m20.157929639s) [upload=699478]: Got next segment. Inline: false Oct 26 16:43:34.695871200 (795h12m20.157963674s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 26 16:43:34.695892140 (795h12m20.157984623s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 26 16:43:34.966350635 (795h12m20.428443108s) [seg_pos={0 0}, upload=699478]: Begin upload segment... Oct 26 16:43:34.966406350 (795h12m20.428498824s) [seg_pos={0 0}, upload=699478]: Done begin upload segment. Oct 26 16:43:34.966430397 (795h12m20.42852288s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.966856137 (795h12m20.42894861s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967260588 (795h12m20.429353061s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967554116 (795h12m20.429646599s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.967816446 (795h12m20.429908919s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968072593 (795h12m20.430165066s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.968318842 (795h12m20.430411325s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.969931955 (795h12m20.432024428s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970272352 (795h12m20.432364825s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.970700378 (795h12m20.432792851s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.971659183 (795h12m20.433751656s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972019149 (795h12m20.434111622s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972173212 (795h12m20.434265685s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.972539679 (795h12m20.434632162s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973297312 (795h12m20.435389795s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:34.973853531 (795h12m20.435946004s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Uploading piece... Oct 26 16:43:36.690556090 (795h12m22.152648583s) [upload=699478]: Next returned nil segment Oct 26 16:43:36.690582791 (795h12m22.152675264s) [upload=699478]: Waiting for error group managing segments... Oct 26 16:43:36.966013153 (795h12m22.428105626s) [noise=true , address=145.79.227.14:4839 , piece_id=I66NXTZCRN52PHNNVV3IR6VPO6OYMG42XTPU2TPHKHXD6IV6OUNQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.967074114 (795h12m22.429166597s) [noise=true , address=145.79.227.13:4801 , piece_id=SKHQ37F63YCNZBTFKUAUAQJFGBOD3UAIM4ZCPVULVLPQ37CK2R2A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.980904178 (795h12m22.442996661s) [noise=true , address=145.79.227.3:4151 , piece_id=DJT2QSWJLOVI2NO52FLSNIIYU7CPAK4UKVYGP4IMRWP2XALBVI6A, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.986250809 (795h12m22.448343292s) [noise=true , address=145.79.227.18:4007 , piece_id=FCNBRI3JL2V4ITYKZLUGSMNFL2R4LJXDADWUK442DD4DPD565WDQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:36.996367744 (795h12m22.458460217s) [noise=true , address=145.79.227.2:4115 , piece_id=6HZ54U7DFT2ZVY6F4OBA33FOMCAT5LPNUEHU2Z7VSNAXBV5X2MSA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.001995259 (795h12m22.464087742s) [noise=true , address=145.79.227.18:4005 , piece_id=AX7UYUK3RBH6ICAB2YEP24WSDW5BII5JVGR7XGDX6NSFDCNDCLMA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.004881014 (795h12m22.466973487s) [noise=true , address=145.79.227.22:4509 , piece_id=PCQTTTMV5BXNKMH7SXMUSV42JETSBPXNZ6C2FZ4PKY2IHNLWYMLQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.014319087 (795h12m22.47641157s) [noise=true , address=145.79.227.5:4222 , piece_id=GEALWLL7PSLCUCRHLYO5OD7VZOGPBE5OAUDNMNMDTA43X6EQU5UQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.015925896 (795h12m22.478018369s) [noise=true , address=145.79.227.4:4186 , piece_id=NS7GIXORICTXIRWJLFDH2OK3VK7WIRLXMB2OWS7NJR5M6PCKF6EQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.078750817 (795h12m22.5408433s) [noise=true , address=145.79.227.19:4079 , piece_id=T4HVSBHNAXX4H4URLR4F4NUDYGBMXU3I4QRGS5TAFJYJJZYBD5UA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.092225525 (795h12m22.554317998s) [noise=true , address=145.79.227.23:4546 , piece_id=LCFU4IOGACZ64AMAEZVOSND5J7XOJRH3ZUN3276RVDE2H6R35YJQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.224430600 (795h12m22.686523083s) [noise=true , address=77.90.31.13:4626 , piece_id=4HYFPY23KJKJOB6PDQCJBMF2I6P3ITMB6IRS7KCLX75GM7PLG3GQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252860535 (795h12m22.714953009s) [noise=true , address=77.90.31.14:4663 , piece_id=JHEIRQXRTKLFQVYSV4MZH473FKYC5N4XYM6DXAJV5TFWFB5JZC3Q, seg_pos={0 0}, upload=699478]: Done uploading piece. err: <nil> Oct 26 16:43:37.252903397 (795h12m22.71499587s) [seg_pos={0 0}, upload=699478]: Segment reached optimal threshold of 13 pieces. Oct 26 16:43:37.253093449 (795h12m22.715185942s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Oct 26 16:43:37.253104831 (795h12m22.715197314s) [noise=true , address=45.91.116.4:4940 , piece_id=OWYYZ7TVTSYF7B3AMNVYAA4F43XHZM4XRZ4YD7YH7WXM47CDXGEQ, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253276057 (795h12m22.71536854s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: context canceled; context canceled Oct 26 16:43:37.253287398 (795h12m22.715379881s) [noise=true , address=77.90.31.11:4555 , piece_id=QSRWXDFFQOAFDUGK53BJQEC5GA2IPVPGC6QGF3SBJE2DBBIAVNQA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253378742 (795h12m22.715471215s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: read tcp 188.166.225.170:49922->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 26 16:43:37.253403239 (795h12m22.715495712s) [noise=true , address=77.90.31.16:4734 , piece_id=SVST4SBUXF2TOWIFUB44NYDJHAW3OI7D2P3MKBD7K4AY6CG5F7RA, seg_pos={0 0}, upload=699478]: Long tail cancellation due to optimal threshold Oct 26 16:43:37.253430731 (795h12m22.715523204s) [seg_pos={0 0}, upload=699478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 26 16:43:37.253526223 (795h12m22.715618706s) [upload=699478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 26 16:43:37.253534318 (795h12m22.715626791s) [upload=699478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 26 16:43:37.557381411 (795h12m23.019473894s) [upload=699478]: Done uploading segments... Oct 26 16:43:37.557409194 (795h12m23.019501667s) [upload=699478]: Upload finished. err: <nil> Duration: 30.002939713s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.58989449s FAIL (2025-10-26 15:55:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.589940227s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.887876241s FAIL (2025-10-26 15:54:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.887901428s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.74004963s FAIL (2025-10-26 15:53:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.740074276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.693318248s FAIL (2025-10-26 15:52:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.69334572s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)