Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX....X...X..X] duration: 30.003028667s trace: https://jaeger.dev.storj.io/trace/25dc0d63792f98b7 FAIL (2025-12-28 01:15:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: VNLZCGXLWLEPTDU7BRQVJOVAWTOCRYSVGGHNEHK7IBAP7JZURDCA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: WZHCETWXGTVOYZCC3L2YH6D6ATO7GWBFZI4IKXLLNC3WL2EHTEOQ) context deadline exceeded ======LOGS====== .. Dec 28 01:14:40.377062163 (537h45m40.248145198s) [upload=471621]: Got next segment. Inline: false Dec 28 01:14:40.377099504 (537h45m40.248182539s) [upload=471621]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:14:40.377105345 (537h45m40.24818838s) [upload=471621]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:14:40.617689012 (537h45m40.488772047s) [seg_pos={0 0}, upload=471621]: Begin upload segment... Dec 28 01:14:40.617741531 (537h45m40.488824566s) [seg_pos={0 0}, upload=471621]: Done begin upload segment. Dec 28 01:14:40.617782890 (537h45m40.488865925s) [noise=true , address=145.79.227.12:4765 , piece_id=VSCZVBZHLP3AJDREHY66NARYBDVYAIQNGF2PBI4MVRAOYCX7JORA, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.617876198 (537h45m40.488959243s) [noise=true , address=145.79.227.19:4077 , piece_id=VA5EH6UHCR25GX6S42BZ2A5HWYROOIZFWI5XK6PN2VTROHU2PXVA, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.618702992 (537h45m40.489786037s) [noise=true , address=145.79.227.15:4876 , piece_id=O6DU7UY5KL5RS6Z3JP2JPGW6S5NZN44DFNUVVXYFNVTL5D3JWPQQ, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.619210638 (537h45m40.490293673s) [noise=true , address=145.79.227.29:4943 , piece_id=TAOEGV3NE2EVQZWOSIVAWJU47YJD72YS35462VZSM6KR7FB25WIQ, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.619740397 (537h45m40.490823442s) [noise=true , address=45.91.116.10:4083 , piece_id=WZHCETWXGTVOYZCC3L2YH6D6ATO7GWBFZI4IKXLLNC3WL2EHTEOQ, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.620278692 (537h45m40.491361737s) [noise=true , address=145.79.227.16:4910 , piece_id=Z7LN225QTG7PZ4RJIFQYQQ5WRTPG2OO2R5SZ5XXO33HUOF2SIOKA, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.621506440 (537h45m40.492589475s) [noise=true , address=45.91.116.11:4119 , piece_id=RT3C6FPV3XZYEX2NVSGKGNZ7ZMKSASP3BOPXTNKA6TQA3CGYY7GQ, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.621946087 (537h45m40.493029122s) [noise=true , address=145.79.227.24:4581 , piece_id=WHVODLYK5HVFVARFEKZMFB7ADGIADAXTHREOAZ3WL2K7MWKLC32Q, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.622788311 (537h45m40.493871346s) [noise=true , address=45.91.116.8:4046 , piece_id=VNLZCGXLWLEPTDU7BRQVJOVAWTOCRYSVGGHNEHK7IBAP7JZURDCA, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.623800979 (537h45m40.494884024s) [noise=true , address=145.79.227.31:4798 , piece_id=VEIQ22O6N63KVJNH7FL7KJH7ZIQHS6TD7JALBX3IUA5AD4UU5RIA, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.624283398 (537h45m40.495366443s) [noise=true , address=145.79.227.9:4367 , piece_id=ZGL6DBDE4BGQXMCI32LUXJILBYN7T4XIUDGSTBIY3EGJFFKP2DRA, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.625644039 (537h45m40.496727084s) [noise=true , address=145.79.227.7:4295 , piece_id=CB2T7647EEM3ABOMTEVABTXHY56L4WIPPNXENFEEGMQZDEWMNM5A, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.625674116 (537h45m40.496757151s) [noise=true , address=77.90.31.10:4520 , piece_id=HB2477TWDCSFSFBT5BAKZJJ6LU44MYVXJ56VWWJIZ6Y2CUD56UKA, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.627041029 (537h45m40.498124064s) [noise=true , address=145.79.227.46:4603 , piece_id=HTJHOICYE5R7YOHNW5UL4MPIDHHIHBCGBWUKJC4C6TIDIQINXDUQ, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.630033516 (537h45m40.501116551s) [noise=true , address=77.90.31.21:4878 , piece_id=CU547PV7MUTMFCBGAAQS7VIXPVJEOJRC77N5FPHN24DOUBBD2PDA, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:40.631877938 (537h45m40.502960973s) [noise=true , address=145.79.227.11:4729 , piece_id=PLTLX2GK4PYPP3YHW6QX2WVSWW4ZSFWPTKTUHNVTRFIOWTRHNDTA, seg_pos={0 0}, upload=471621]: Uploading piece... Dec 28 01:14:42.232934320 (537h45m42.104017355s) [upload=471621]: Next returned nil segment Dec 28 01:14:42.232948918 (537h45m42.104031963s) [upload=471621]: Waiting for error group managing segments... Dec 28 01:14:42.621692578 (537h45m42.492775613s) [noise=true , address=145.79.227.31:4798 , piece_id=VEIQ22O6N63KVJNH7FL7KJH7ZIQHS6TD7JALBX3IUA5AD4UU5RIA, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.633265014 (537h45m42.504348059s) [noise=true , address=145.79.227.7:4295 , piece_id=CB2T7647EEM3ABOMTEVABTXHY56L4WIPPNXENFEEGMQZDEWMNM5A, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.633325758 (537h45m42.504408794s) [noise=true , address=145.79.227.11:4729 , piece_id=PLTLX2GK4PYPP3YHW6QX2WVSWW4ZSFWPTKTUHNVTRFIOWTRHNDTA, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.646235059 (537h45m42.517318094s) [noise=true , address=145.79.227.46:4603 , piece_id=HTJHOICYE5R7YOHNW5UL4MPIDHHIHBCGBWUKJC4C6TIDIQINXDUQ, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.648861540 (537h45m42.519944575s) [noise=true , address=145.79.227.19:4077 , piece_id=VA5EH6UHCR25GX6S42BZ2A5HWYROOIZFWI5XK6PN2VTROHU2PXVA, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.651146441 (537h45m42.522229476s) [noise=true , address=145.79.227.9:4367 , piece_id=ZGL6DBDE4BGQXMCI32LUXJILBYN7T4XIUDGSTBIY3EGJFFKP2DRA, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.667657550 (537h45m42.538740585s) [noise=true , address=145.79.227.24:4581 , piece_id=WHVODLYK5HVFVARFEKZMFB7ADGIADAXTHREOAZ3WL2K7MWKLC32Q, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.831839741 (537h45m42.702922776s) [noise=true , address=45.91.116.8:4046 , piece_id=VNLZCGXLWLEPTDU7BRQVJOVAWTOCRYSVGGHNEHK7IBAP7JZURDCA, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.855131691 (537h45m42.726214726s) [noise=true , address=45.91.116.11:4119 , piece_id=RT3C6FPV3XZYEX2NVSGKGNZ7ZMKSASP3BOPXTNKA6TQA3CGYY7GQ, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.859415968 (537h45m42.730499014s) [noise=true , address=45.91.116.10:4083 , piece_id=WZHCETWXGTVOYZCC3L2YH6D6ATO7GWBFZI4IKXLLNC3WL2EHTEOQ, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.867855578 (537h45m42.738938624s) [noise=true , address=145.79.227.12:4765 , piece_id=VSCZVBZHLP3AJDREHY66NARYBDVYAIQNGF2PBI4MVRAOYCX7JORA, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:42.888972687 (537h45m42.760055723s) [noise=true , address=77.90.31.10:4520 , piece_id=HB2477TWDCSFSFBT5BAKZJJ6LU44MYVXJ56VWWJIZ6Y2CUD56UKA, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:43.063337207 (537h45m42.934420252s) [noise=true , address=145.79.227.15:4876 , piece_id=O6DU7UY5KL5RS6Z3JP2JPGW6S5NZN44DFNUVVXYFNVTL5D3JWPQQ, seg_pos={0 0}, upload=471621]: Done uploading piece. err: <nil> Dec 28 01:14:43.063356343 (537h45m42.934439378s) [seg_pos={0 0}, upload=471621]: Segment reached optimal threshold of 13 pieces. Dec 28 01:14:43.063589176 (537h45m42.934672211s) [noise=true , address=145.79.227.16:4910 , piece_id=Z7LN225QTG7PZ4RJIFQYQQ5WRTPG2OO2R5SZ5XXO33HUOF2SIOKA, seg_pos={0 0}, upload=471621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:42366->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:42366->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42366->145.79.227.16:4910: use of closed network connection Dec 28 01:14:43.063595949 (537h45m42.934678984s) [noise=true , address=145.79.227.16:4910 , piece_id=Z7LN225QTG7PZ4RJIFQYQQ5WRTPG2OO2R5SZ5XXO33HUOF2SIOKA, seg_pos={0 0}, upload=471621]: Long tail cancellation due to optimal threshold Dec 28 01:14:43.063794928 (537h45m42.934877973s) [noise=true , address=145.79.227.29:4943 , piece_id=TAOEGV3NE2EVQZWOSIVAWJU47YJD72YS35462VZSM6KR7FB25WIQ, seg_pos={0 0}, upload=471621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 28 01:14:43.063814234 (537h45m42.934897279s) [noise=true , address=145.79.227.29:4943 , piece_id=TAOEGV3NE2EVQZWOSIVAWJU47YJD72YS35462VZSM6KR7FB25WIQ, seg_pos={0 0}, upload=471621]: Long tail cancellation due to optimal threshold Dec 28 01:14:43.063838331 (537h45m42.934921366s) [noise=true , address=77.90.31.21:4878 , piece_id=CU547PV7MUTMFCBGAAQS7VIXPVJEOJRC77N5FPHN24DOUBBD2PDA, seg_pos={0 0}, upload=471621]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:37506->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:37506->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37506->77.90.31.21:4878: use of closed network connection Dec 28 01:14:43.063850083 (537h45m42.934933128s) [noise=true , address=77.90.31.21:4878 , piece_id=CU547PV7MUTMFCBGAAQS7VIXPVJEOJRC77N5FPHN24DOUBBD2PDA, seg_pos={0 0}, upload=471621]: Long tail cancellation due to optimal threshold Dec 28 01:14:43.063881072 (537h45m42.934964107s) [seg_pos={0 0}, upload=471621]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:14:43.063984940 (537h45m42.935067975s) [upload=471621]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:14:43.063990770 (537h45m42.935073806s) [upload=471621]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:14:43.356222521 (537h45m43.227305556s) [upload=471621]: Done uploading segments... Dec 28 01:14:43.356241538 (537h45m43.227324573s) [upload=471621]: Upload finished. err: <nil> Duration: 30.003055308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX....X...X..] duration: 15.975894017s trace: https://jaeger.dev.storj.io/trace/93ba19ea1c02cbb FAIL (2025-12-28 01:13:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: IQGBKCVBFIMV6NJURZ26OZCKTIIFTWAPEBRZD3G67K7BED375JBA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 5X3HZ5WYU7JBCYRISPZDO5UMOME4YKQ6UGOV66CGEKGFME6FZNVA) context deadline exceeded ======LOGS====== Dec 28 01:12:05.029672423 (537h43m4.900755518s) [upload=471584]: Starting upload Dec 28 01:12:05.029763005 (537h43m4.90084604s) [upload=471584]: Done starting upload Dec 28 01:12:05.030029873 (537h43m4.901112908s) [upload=471584]: Uploading segments... Dec 28 01:12:05.030136917 (537h43m4.901219962s) [upload=471584]: Got next segment. Inline: false Dec 28 01:12:05.030199997 (537h43m4.901283042s) [upload=471584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:12:05.030210677 (537h43m4.901293742s) [upload=471584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:12:05.275009571 (537h43m5.146092616s) [seg_pos={0 0}, upload=471584]: Begin upload segment... Dec 28 01:12:05.275060768 (537h43m5.146143813s) [seg_pos={0 0}, upload=471584]: Done begin upload segment. Dec 28 01:12:05.275099582 (537h43m5.146182627s) [noise=true , address=145.79.227.51:4410 , piece_id=CA4FV7GLJFNEDHFPW2FZZ53625VTNX4JJMLHM6WWP6VANB3X76GA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.275108991 (537h43m5.146192026s) [noise=true , address=145.79.227.9:4367 , piece_id=JM6FCUQTB7RCWBYYWMHXV6KQOGMYWS5OPKX5OTEIFOT3WMH4THQQ, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.275538047 (537h43m5.146621082s) [noise=true , address=45.91.116.5:4298 , piece_id=XCTAZNS3HKAN2KN776RSPCLAHPVZQ6PFZZDXNXIOANZ3ZHKMWVRA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.275852366 (537h43m5.146935411s) [noise=true , address=45.91.116.11:4118 , piece_id=IQGBKCVBFIMV6NJURZ26OZCKTIIFTWAPEBRZD3G67K7BED375JBA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.276140174 (537h43m5.147223209s) [noise=true , address=145.79.227.7:4293 , piece_id=EPUXT7FK4RY2IG6WCMGH53745N7PLBFMPR46AEVCXEF4O6657R3Q, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.276459351 (537h43m5.147542386s) [noise=true , address=145.79.227.12:4767 , piece_id=EOTQP4QECYYBEGH4WGPQWTZ2QIO7FSFNXEMCDRY6WDUMFUMWMBFA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.276830228 (537h43m5.147913273s) [noise=true , address=145.79.227.15:4876 , piece_id=CVZMNRK3FKHKG47GAIZKM4IHH7R437ES5KBH7IMTLNXJLIYZ6PFA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.277154535 (537h43m5.14823758s) [noise=true , address=145.79.227.45:4231 , piece_id=3P2EEX3LHTMSCGMVUYUXTCG633TW36SGIX4J44KCYE6LBX5L65KQ, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.277494392 (537h43m5.148577437s) [noise=true , address=145.79.227.31:4798 , piece_id=HW3PJDHQFEAPO56S5W3XTKUI2Z42FZW5WJ2XHZULSLZNPFHWHLOA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.277884295 (537h43m5.14896734s) [noise=true , address=45.91.116.12:4226 , piece_id=WWYHY2XJHKIJH5OVBMQC2X4KBDQHUJKT3S7RD7RYS7AOWHWETVFQ, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.278233039 (537h43m5.149316084s) [noise=true , address=145.79.227.17:4970 , piece_id=5OQKCSZDXCVT6YVMPXOYZDCGTI4A2E47AFWV4Z5KK3GZH7O5IPIA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.279570505 (537h43m5.15065355s) [noise=true , address=45.91.116.8:4046 , piece_id=5X3HZ5WYU7JBCYRISPZDO5UMOME4YKQ6UGOV66CGEKGFME6FZNVA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.279644657 (537h43m5.150727692s) [noise=true , address=145.79.227.38:4979 , piece_id=OVKYHLGQPQQKKHRIJHLXLQ7P6FYRA26SZKL4HYDDMCTUWSVAQ4RA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.281056535 (537h43m5.15213958s) [noise=true , address=45.91.116.9:4261 , piece_id=2T66GAWKFTBPWAJBDYI3JH7NVTSZ6OKTVBIGLI35PV67DBZDAV6A, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.282831605 (537h43m5.15391465s) [noise=true , address=145.79.227.48:4303 , piece_id=XLCCMO23INZF2EJBV4BF6JS6MNWF5U6RCSFUZBCOPSPEDW6QZZ4A, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.287070977 (537h43m5.158154022s) [noise=true , address=77.90.31.11:4555 , piece_id=53GLSTXK25VNTQURRDYVJY3S2HWE4IADWH4DTQK2OTULDTWVUHGA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:06.888741499 (537h43m6.759824544s) [upload=471584]: Next returned nil segment Dec 28 01:12:06.888759384 (537h43m6.759842419s) [upload=471584]: Waiting for error group managing segments... Dec 28 01:12:07.281192248 (537h43m7.152275294s) [noise=true , address=145.79.227.12:4767 , piece_id=EOTQP4QECYYBEGH4WGPQWTZ2QIO7FSFNXEMCDRY6WDUMFUMWMBFA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.296910517 (537h43m7.167993562s) [noise=true , address=145.79.227.7:4293 , piece_id=EPUXT7FK4RY2IG6WCMGH53745N7PLBFMPR46AEVCXEF4O6657R3Q, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.302106812 (537h43m7.173189847s) [noise=true , address=145.79.227.31:4798 , piece_id=HW3PJDHQFEAPO56S5W3XTKUI2Z42FZW5WJ2XHZULSLZNPFHWHLOA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.303925685 (537h43m7.17500872s) [noise=true , address=145.79.227.51:4410 , piece_id=CA4FV7GLJFNEDHFPW2FZZ53625VTNX4JJMLHM6WWP6VANB3X76GA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.306184626 (537h43m7.177267671s) [noise=true , address=145.79.227.38:4979 , piece_id=OVKYHLGQPQQKKHRIJHLXLQ7P6FYRA26SZKL4HYDDMCTUWSVAQ4RA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.308299965 (537h43m7.17938301s) [noise=true , address=145.79.227.48:4303 , piece_id=XLCCMO23INZF2EJBV4BF6JS6MNWF5U6RCSFUZBCOPSPEDW6QZZ4A, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.308567143 (537h43m7.179650188s) [noise=true , address=145.79.227.45:4231 , piece_id=3P2EEX3LHTMSCGMVUYUXTCG633TW36SGIX4J44KCYE6LBX5L65KQ, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.308962306 (537h43m7.180045341s) [noise=true , address=145.79.227.17:4970 , piece_id=5OQKCSZDXCVT6YVMPXOYZDCGTI4A2E47AFWV4Z5KK3GZH7O5IPIA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.314574943 (537h43m7.185657978s) [noise=true , address=145.79.227.15:4876 , piece_id=CVZMNRK3FKHKG47GAIZKM4IHH7R437ES5KBH7IMTLNXJLIYZ6PFA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.316736980 (537h43m7.187820015s) [noise=true , address=145.79.227.9:4367 , piece_id=JM6FCUQTB7RCWBYYWMHXV6KQOGMYWS5OPKX5OTEIFOT3WMH4THQQ, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.506868450 (537h43m7.377951485s) [noise=true , address=45.91.116.8:4046 , piece_id=5X3HZ5WYU7JBCYRISPZDO5UMOME4YKQ6UGOV66CGEKGFME6FZNVA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.528524535 (537h43m7.39960757s) [noise=true , address=45.91.116.11:4118 , piece_id=IQGBKCVBFIMV6NJURZ26OZCKTIIFTWAPEBRZD3G67K7BED375JBA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.533403044 (537h43m7.404486089s) [noise=true , address=45.91.116.5:4298 , piece_id=XCTAZNS3HKAN2KN776RSPCLAHPVZQ6PFZZDXNXIOANZ3ZHKMWVRA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.533429114 (537h43m7.404512159s) [seg_pos={0 0}, upload=471584]: Segment reached optimal threshold of 13 pieces. Dec 28 01:12:07.533659443 (537h43m7.404742498s) [noise=true , address=45.91.116.9:4261 , piece_id=2T66GAWKFTBPWAJBDYI3JH7NVTSZ6OKTVBIGLI35PV67DBZDAV6A, seg_pos={0 0}, upload=471584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 28 01:12:07.533671215 (537h43m7.40475426s) [noise=true , address=45.91.116.9:4261 , piece_id=2T66GAWKFTBPWAJBDYI3JH7NVTSZ6OKTVBIGLI35PV67DBZDAV6A, seg_pos={0 0}, upload=471584]: Long tail cancellation due to optimal threshold Dec 28 01:12:07.533829277 (537h43m7.404912322s) [noise=true , address=45.91.116.12:4226 , piece_id=WWYHY2XJHKIJH5OVBMQC2X4KBDQHUJKT3S7RD7RYS7AOWHWETVFQ, seg_pos={0 0}, upload=471584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 01:12:07.533844896 (537h43m7.404927941s) [noise=true , address=45.91.116.12:4226 , piece_id=WWYHY2XJHKIJH5OVBMQC2X4KBDQHUJKT3S7RD7RYS7AOWHWETVFQ, seg_pos={0 0}, upload=471584]: Long tail cancellation due to optimal threshold Dec 28 01:12:07.534004871 (537h43m7.405087916s) [noise=true , address=77.90.31.11:4555 , piece_id=53GLSTXK25VNTQURRDYVJY3S2HWE4IADWH4DTQK2OTULDTWVUHGA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:46296->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:46296->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46296->77.90.31.11:4555: use of closed network connection Dec 28 01:12:07.534011874 (537h43m7.405094919s) [noise=true , address=77.90.31.11:4555 , piece_id=53GLSTXK25VNTQURRDYVJY3S2HWE4IADWH4DTQK2OTULDTWVUHGA, seg_pos={0 0}, upload=471584]: Long tail cancellation due to optimal threshold Dec 28 01:12:07.534034858 (537h43m7.405117893s) [seg_pos={0 0}, upload=471584]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:12:07.534121312 (537h43m7.405204347s) [upload=471584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:12:07.534125721 (537h43m7.405208756s) [upload=471584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:12:07.824127004 (537h43m7.695210039s) [upload=471584]: Done uploading segments... Dec 28 01:12:07.824150890 (537h43m7.695233935s) [upload=471584]: Upload finished. err: <nil> Duration: 15.975914816s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX....X...X.] duration: 26.79582536s trace: https://jaeger.dev.storj.io/trace/93ba19ea1c02cbb FAIL (2025-12-28 01:13:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: IQGBKCVBFIMV6NJURZ26OZCKTIIFTWAPEBRZD3G67K7BED375JBA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 5X3HZ5WYU7JBCYRISPZDO5UMOME4YKQ6UGOV66CGEKGFME6FZNVA) context deadline exceeded ======LOGS====== Dec 28 01:12:05.029672423 (537h43m4.900755518s) [upload=471584]: Starting upload Dec 28 01:12:05.029763005 (537h43m4.90084604s) [upload=471584]: Done starting upload Dec 28 01:12:05.030029873 (537h43m4.901112908s) [upload=471584]: Uploading segments... Dec 28 01:12:05.030136917 (537h43m4.901219962s) [upload=471584]: Got next segment. Inline: false Dec 28 01:12:05.030199997 (537h43m4.901283042s) [upload=471584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:12:05.030210677 (537h43m4.901293742s) [upload=471584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:12:05.275009571 (537h43m5.146092616s) [seg_pos={0 0}, upload=471584]: Begin upload segment... Dec 28 01:12:05.275060768 (537h43m5.146143813s) [seg_pos={0 0}, upload=471584]: Done begin upload segment. Dec 28 01:12:05.275099582 (537h43m5.146182627s) [noise=true , address=145.79.227.51:4410 , piece_id=CA4FV7GLJFNEDHFPW2FZZ53625VTNX4JJMLHM6WWP6VANB3X76GA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.275108991 (537h43m5.146192026s) [noise=true , address=145.79.227.9:4367 , piece_id=JM6FCUQTB7RCWBYYWMHXV6KQOGMYWS5OPKX5OTEIFOT3WMH4THQQ, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.275538047 (537h43m5.146621082s) [noise=true , address=45.91.116.5:4298 , piece_id=XCTAZNS3HKAN2KN776RSPCLAHPVZQ6PFZZDXNXIOANZ3ZHKMWVRA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.275852366 (537h43m5.146935411s) [noise=true , address=45.91.116.11:4118 , piece_id=IQGBKCVBFIMV6NJURZ26OZCKTIIFTWAPEBRZD3G67K7BED375JBA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.276140174 (537h43m5.147223209s) [noise=true , address=145.79.227.7:4293 , piece_id=EPUXT7FK4RY2IG6WCMGH53745N7PLBFMPR46AEVCXEF4O6657R3Q, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.276459351 (537h43m5.147542386s) [noise=true , address=145.79.227.12:4767 , piece_id=EOTQP4QECYYBEGH4WGPQWTZ2QIO7FSFNXEMCDRY6WDUMFUMWMBFA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.276830228 (537h43m5.147913273s) [noise=true , address=145.79.227.15:4876 , piece_id=CVZMNRK3FKHKG47GAIZKM4IHH7R437ES5KBH7IMTLNXJLIYZ6PFA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.277154535 (537h43m5.14823758s) [noise=true , address=145.79.227.45:4231 , piece_id=3P2EEX3LHTMSCGMVUYUXTCG633TW36SGIX4J44KCYE6LBX5L65KQ, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.277494392 (537h43m5.148577437s) [noise=true , address=145.79.227.31:4798 , piece_id=HW3PJDHQFEAPO56S5W3XTKUI2Z42FZW5WJ2XHZULSLZNPFHWHLOA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.277884295 (537h43m5.14896734s) [noise=true , address=45.91.116.12:4226 , piece_id=WWYHY2XJHKIJH5OVBMQC2X4KBDQHUJKT3S7RD7RYS7AOWHWETVFQ, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.278233039 (537h43m5.149316084s) [noise=true , address=145.79.227.17:4970 , piece_id=5OQKCSZDXCVT6YVMPXOYZDCGTI4A2E47AFWV4Z5KK3GZH7O5IPIA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.279570505 (537h43m5.15065355s) [noise=true , address=45.91.116.8:4046 , piece_id=5X3HZ5WYU7JBCYRISPZDO5UMOME4YKQ6UGOV66CGEKGFME6FZNVA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.279644657 (537h43m5.150727692s) [noise=true , address=145.79.227.38:4979 , piece_id=OVKYHLGQPQQKKHRIJHLXLQ7P6FYRA26SZKL4HYDDMCTUWSVAQ4RA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.281056535 (537h43m5.15213958s) [noise=true , address=45.91.116.9:4261 , piece_id=2T66GAWKFTBPWAJBDYI3JH7NVTSZ6OKTVBIGLI35PV67DBZDAV6A, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.282831605 (537h43m5.15391465s) [noise=true , address=145.79.227.48:4303 , piece_id=XLCCMO23INZF2EJBV4BF6JS6MNWF5U6RCSFUZBCOPSPEDW6QZZ4A, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.287070977 (537h43m5.158154022s) [noise=true , address=77.90.31.11:4555 , piece_id=53GLSTXK25VNTQURRDYVJY3S2HWE4IADWH4DTQK2OTULDTWVUHGA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:06.888741499 (537h43m6.759824544s) [upload=471584]: Next returned nil segment Dec 28 01:12:06.888759384 (537h43m6.759842419s) [upload=471584]: Waiting for error group managing segments... Dec 28 01:12:07.281192248 (537h43m7.152275294s) [noise=true , address=145.79.227.12:4767 , piece_id=EOTQP4QECYYBEGH4WGPQWTZ2QIO7FSFNXEMCDRY6WDUMFUMWMBFA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.296910517 (537h43m7.167993562s) [noise=true , address=145.79.227.7:4293 , piece_id=EPUXT7FK4RY2IG6WCMGH53745N7PLBFMPR46AEVCXEF4O6657R3Q, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.302106812 (537h43m7.173189847s) [noise=true , address=145.79.227.31:4798 , piece_id=HW3PJDHQFEAPO56S5W3XTKUI2Z42FZW5WJ2XHZULSLZNPFHWHLOA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.303925685 (537h43m7.17500872s) [noise=true , address=145.79.227.51:4410 , piece_id=CA4FV7GLJFNEDHFPW2FZZ53625VTNX4JJMLHM6WWP6VANB3X76GA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.306184626 (537h43m7.177267671s) [noise=true , address=145.79.227.38:4979 , piece_id=OVKYHLGQPQQKKHRIJHLXLQ7P6FYRA26SZKL4HYDDMCTUWSVAQ4RA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.308299965 (537h43m7.17938301s) [noise=true , address=145.79.227.48:4303 , piece_id=XLCCMO23INZF2EJBV4BF6JS6MNWF5U6RCSFUZBCOPSPEDW6QZZ4A, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.308567143 (537h43m7.179650188s) [noise=true , address=145.79.227.45:4231 , piece_id=3P2EEX3LHTMSCGMVUYUXTCG633TW36SGIX4J44KCYE6LBX5L65KQ, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.308962306 (537h43m7.180045341s) [noise=true , address=145.79.227.17:4970 , piece_id=5OQKCSZDXCVT6YVMPXOYZDCGTI4A2E47AFWV4Z5KK3GZH7O5IPIA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.314574943 (537h43m7.185657978s) [noise=true , address=145.79.227.15:4876 , piece_id=CVZMNRK3FKHKG47GAIZKM4IHH7R437ES5KBH7IMTLNXJLIYZ6PFA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.316736980 (537h43m7.187820015s) [noise=true , address=145.79.227.9:4367 , piece_id=JM6FCUQTB7RCWBYYWMHXV6KQOGMYWS5OPKX5OTEIFOT3WMH4THQQ, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.506868450 (537h43m7.377951485s) [noise=true , address=45.91.116.8:4046 , piece_id=5X3HZ5WYU7JBCYRISPZDO5UMOME4YKQ6UGOV66CGEKGFME6FZNVA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.528524535 (537h43m7.39960757s) [noise=true , address=45.91.116.11:4118 , piece_id=IQGBKCVBFIMV6NJURZ26OZCKTIIFTWAPEBRZD3G67K7BED375JBA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.533403044 (537h43m7.404486089s) [noise=true , address=45.91.116.5:4298 , piece_id=XCTAZNS3HKAN2KN776RSPCLAHPVZQ6PFZZDXNXIOANZ3ZHKMWVRA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.533429114 (537h43m7.404512159s) [seg_pos={0 0}, upload=471584]: Segment reached optimal threshold of 13 pieces. Dec 28 01:12:07.533659443 (537h43m7.404742498s) [noise=true , address=45.91.116.9:4261 , piece_id=2T66GAWKFTBPWAJBDYI3JH7NVTSZ6OKTVBIGLI35PV67DBZDAV6A, seg_pos={0 0}, upload=471584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 28 01:12:07.533671215 (537h43m7.40475426s) [noise=true , address=45.91.116.9:4261 , piece_id=2T66GAWKFTBPWAJBDYI3JH7NVTSZ6OKTVBIGLI35PV67DBZDAV6A, seg_pos={0 0}, upload=471584]: Long tail cancellation due to optimal threshold Dec 28 01:12:07.533829277 (537h43m7.404912322s) [noise=true , address=45.91.116.12:4226 , piece_id=WWYHY2XJHKIJH5OVBMQC2X4KBDQHUJKT3S7RD7RYS7AOWHWETVFQ, seg_pos={0 0}, upload=471584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 01:12:07.533844896 (537h43m7.404927941s) [noise=true , address=45.91.116.12:4226 , piece_id=WWYHY2XJHKIJH5OVBMQC2X4KBDQHUJKT3S7RD7RYS7AOWHWETVFQ, seg_pos={0 0}, upload=471584]: Long tail cancellation due to optimal threshold Dec 28 01:12:07.534004871 (537h43m7.405087916s) [noise=true , address=77.90.31.11:4555 , piece_id=53GLSTXK25VNTQURRDYVJY3S2HWE4IADWH4DTQK2OTULDTWVUHGA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:46296->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:46296->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46296->77.90.31.11:4555: use of closed network connection Dec 28 01:12:07.534011874 (537h43m7.405094919s) [noise=true , address=77.90.31.11:4555 , piece_id=53GLSTXK25VNTQURRDYVJY3S2HWE4IADWH4DTQK2OTULDTWVUHGA, seg_pos={0 0}, upload=471584]: Long tail cancellation due to optimal threshold Dec 28 01:12:07.534034858 (537h43m7.405117893s) [seg_pos={0 0}, upload=471584]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:12:07.534121312 (537h43m7.405204347s) [upload=471584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:12:07.534125721 (537h43m7.405208756s) [upload=471584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:12:07.824127004 (537h43m7.695210039s) [upload=471584]: Done uploading segments... Dec 28 01:12:07.824150890 (537h43m7.695233935s) [upload=471584]: Upload finished. err: <nil> Duration: 26.795852691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX....X...X] duration: 30.002875013s trace: https://jaeger.dev.storj.io/trace/93ba19ea1c02cbb FAIL (2025-12-28 01:12:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: IQGBKCVBFIMV6NJURZ26OZCKTIIFTWAPEBRZD3G67K7BED375JBA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 5X3HZ5WYU7JBCYRISPZDO5UMOME4YKQ6UGOV66CGEKGFME6FZNVA) context deadline exceeded ======LOGS====== Dec 28 01:12:05.029672423 (537h43m4.900755518s) [upload=471584]: Starting upload Dec 28 01:12:05.029763005 (537h43m4.90084604s) [upload=471584]: Done starting upload Dec 28 01:12:05.030029873 (537h43m4.901112908s) [upload=471584]: Uploading segments... Dec 28 01:12:05.030136917 (537h43m4.901219962s) [upload=471584]: Got next segment. Inline: false Dec 28 01:12:05.030199997 (537h43m4.901283042s) [upload=471584]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:12:05.030210677 (537h43m4.901293742s) [upload=471584]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:12:05.275009571 (537h43m5.146092616s) [seg_pos={0 0}, upload=471584]: Begin upload segment... Dec 28 01:12:05.275060768 (537h43m5.146143813s) [seg_pos={0 0}, upload=471584]: Done begin upload segment. Dec 28 01:12:05.275099582 (537h43m5.146182627s) [noise=true , address=145.79.227.51:4410 , piece_id=CA4FV7GLJFNEDHFPW2FZZ53625VTNX4JJMLHM6WWP6VANB3X76GA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.275108991 (537h43m5.146192026s) [noise=true , address=145.79.227.9:4367 , piece_id=JM6FCUQTB7RCWBYYWMHXV6KQOGMYWS5OPKX5OTEIFOT3WMH4THQQ, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.275538047 (537h43m5.146621082s) [noise=true , address=45.91.116.5:4298 , piece_id=XCTAZNS3HKAN2KN776RSPCLAHPVZQ6PFZZDXNXIOANZ3ZHKMWVRA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.275852366 (537h43m5.146935411s) [noise=true , address=45.91.116.11:4118 , piece_id=IQGBKCVBFIMV6NJURZ26OZCKTIIFTWAPEBRZD3G67K7BED375JBA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.276140174 (537h43m5.147223209s) [noise=true , address=145.79.227.7:4293 , piece_id=EPUXT7FK4RY2IG6WCMGH53745N7PLBFMPR46AEVCXEF4O6657R3Q, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.276459351 (537h43m5.147542386s) [noise=true , address=145.79.227.12:4767 , piece_id=EOTQP4QECYYBEGH4WGPQWTZ2QIO7FSFNXEMCDRY6WDUMFUMWMBFA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.276830228 (537h43m5.147913273s) [noise=true , address=145.79.227.15:4876 , piece_id=CVZMNRK3FKHKG47GAIZKM4IHH7R437ES5KBH7IMTLNXJLIYZ6PFA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.277154535 (537h43m5.14823758s) [noise=true , address=145.79.227.45:4231 , piece_id=3P2EEX3LHTMSCGMVUYUXTCG633TW36SGIX4J44KCYE6LBX5L65KQ, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.277494392 (537h43m5.148577437s) [noise=true , address=145.79.227.31:4798 , piece_id=HW3PJDHQFEAPO56S5W3XTKUI2Z42FZW5WJ2XHZULSLZNPFHWHLOA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.277884295 (537h43m5.14896734s) [noise=true , address=45.91.116.12:4226 , piece_id=WWYHY2XJHKIJH5OVBMQC2X4KBDQHUJKT3S7RD7RYS7AOWHWETVFQ, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.278233039 (537h43m5.149316084s) [noise=true , address=145.79.227.17:4970 , piece_id=5OQKCSZDXCVT6YVMPXOYZDCGTI4A2E47AFWV4Z5KK3GZH7O5IPIA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.279570505 (537h43m5.15065355s) [noise=true , address=45.91.116.8:4046 , piece_id=5X3HZ5WYU7JBCYRISPZDO5UMOME4YKQ6UGOV66CGEKGFME6FZNVA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.279644657 (537h43m5.150727692s) [noise=true , address=145.79.227.38:4979 , piece_id=OVKYHLGQPQQKKHRIJHLXLQ7P6FYRA26SZKL4HYDDMCTUWSVAQ4RA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.281056535 (537h43m5.15213958s) [noise=true , address=45.91.116.9:4261 , piece_id=2T66GAWKFTBPWAJBDYI3JH7NVTSZ6OKTVBIGLI35PV67DBZDAV6A, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.282831605 (537h43m5.15391465s) [noise=true , address=145.79.227.48:4303 , piece_id=XLCCMO23INZF2EJBV4BF6JS6MNWF5U6RCSFUZBCOPSPEDW6QZZ4A, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:05.287070977 (537h43m5.158154022s) [noise=true , address=77.90.31.11:4555 , piece_id=53GLSTXK25VNTQURRDYVJY3S2HWE4IADWH4DTQK2OTULDTWVUHGA, seg_pos={0 0}, upload=471584]: Uploading piece... Dec 28 01:12:06.888741499 (537h43m6.759824544s) [upload=471584]: Next returned nil segment Dec 28 01:12:06.888759384 (537h43m6.759842419s) [upload=471584]: Waiting for error group managing segments... Dec 28 01:12:07.281192248 (537h43m7.152275294s) [noise=true , address=145.79.227.12:4767 , piece_id=EOTQP4QECYYBEGH4WGPQWTZ2QIO7FSFNXEMCDRY6WDUMFUMWMBFA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.296910517 (537h43m7.167993562s) [noise=true , address=145.79.227.7:4293 , piece_id=EPUXT7FK4RY2IG6WCMGH53745N7PLBFMPR46AEVCXEF4O6657R3Q, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.302106812 (537h43m7.173189847s) [noise=true , address=145.79.227.31:4798 , piece_id=HW3PJDHQFEAPO56S5W3XTKUI2Z42FZW5WJ2XHZULSLZNPFHWHLOA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.303925685 (537h43m7.17500872s) [noise=true , address=145.79.227.51:4410 , piece_id=CA4FV7GLJFNEDHFPW2FZZ53625VTNX4JJMLHM6WWP6VANB3X76GA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.306184626 (537h43m7.177267671s) [noise=true , address=145.79.227.38:4979 , piece_id=OVKYHLGQPQQKKHRIJHLXLQ7P6FYRA26SZKL4HYDDMCTUWSVAQ4RA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.308299965 (537h43m7.17938301s) [noise=true , address=145.79.227.48:4303 , piece_id=XLCCMO23INZF2EJBV4BF6JS6MNWF5U6RCSFUZBCOPSPEDW6QZZ4A, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.308567143 (537h43m7.179650188s) [noise=true , address=145.79.227.45:4231 , piece_id=3P2EEX3LHTMSCGMVUYUXTCG633TW36SGIX4J44KCYE6LBX5L65KQ, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.308962306 (537h43m7.180045341s) [noise=true , address=145.79.227.17:4970 , piece_id=5OQKCSZDXCVT6YVMPXOYZDCGTI4A2E47AFWV4Z5KK3GZH7O5IPIA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.314574943 (537h43m7.185657978s) [noise=true , address=145.79.227.15:4876 , piece_id=CVZMNRK3FKHKG47GAIZKM4IHH7R437ES5KBH7IMTLNXJLIYZ6PFA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.316736980 (537h43m7.187820015s) [noise=true , address=145.79.227.9:4367 , piece_id=JM6FCUQTB7RCWBYYWMHXV6KQOGMYWS5OPKX5OTEIFOT3WMH4THQQ, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.506868450 (537h43m7.377951485s) [noise=true , address=45.91.116.8:4046 , piece_id=5X3HZ5WYU7JBCYRISPZDO5UMOME4YKQ6UGOV66CGEKGFME6FZNVA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.528524535 (537h43m7.39960757s) [noise=true , address=45.91.116.11:4118 , piece_id=IQGBKCVBFIMV6NJURZ26OZCKTIIFTWAPEBRZD3G67K7BED375JBA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.533403044 (537h43m7.404486089s) [noise=true , address=45.91.116.5:4298 , piece_id=XCTAZNS3HKAN2KN776RSPCLAHPVZQ6PFZZDXNXIOANZ3ZHKMWVRA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: <nil> Dec 28 01:12:07.533429114 (537h43m7.404512159s) [seg_pos={0 0}, upload=471584]: Segment reached optimal threshold of 13 pieces. Dec 28 01:12:07.533659443 (537h43m7.404742498s) [noise=true , address=45.91.116.9:4261 , piece_id=2T66GAWKFTBPWAJBDYI3JH7NVTSZ6OKTVBIGLI35PV67DBZDAV6A, seg_pos={0 0}, upload=471584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 28 01:12:07.533671215 (537h43m7.40475426s) [noise=true , address=45.91.116.9:4261 , piece_id=2T66GAWKFTBPWAJBDYI3JH7NVTSZ6OKTVBIGLI35PV67DBZDAV6A, seg_pos={0 0}, upload=471584]: Long tail cancellation due to optimal threshold Dec 28 01:12:07.533829277 (537h43m7.404912322s) [noise=true , address=45.91.116.12:4226 , piece_id=WWYHY2XJHKIJH5OVBMQC2X4KBDQHUJKT3S7RD7RYS7AOWHWETVFQ, seg_pos={0 0}, upload=471584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 01:12:07.533844896 (537h43m7.404927941s) [noise=true , address=45.91.116.12:4226 , piece_id=WWYHY2XJHKIJH5OVBMQC2X4KBDQHUJKT3S7RD7RYS7AOWHWETVFQ, seg_pos={0 0}, upload=471584]: Long tail cancellation due to optimal threshold Dec 28 01:12:07.534004871 (537h43m7.405087916s) [noise=true , address=77.90.31.11:4555 , piece_id=53GLSTXK25VNTQURRDYVJY3S2HWE4IADWH4DTQK2OTULDTWVUHGA, seg_pos={0 0}, upload=471584]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:46296->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:46296->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46296->77.90.31.11:4555: use of closed network connection Dec 28 01:12:07.534011874 (537h43m7.405094919s) [noise=true , address=77.90.31.11:4555 , piece_id=53GLSTXK25VNTQURRDYVJY3S2HWE4IADWH4DTQK2OTULDTWVUHGA, seg_pos={0 0}, upload=471584]: Long tail cancellation due to optimal threshold Dec 28 01:12:07.534034858 (537h43m7.405117893s) [seg_pos={0 0}, upload=471584]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:12:07.534121312 (537h43m7.405204347s) [upload=471584]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:12:07.534125721 (537h43m7.405208756s) [upload=471584]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:12:07.824127004 (537h43m7.695210039s) [upload=471584]: Done uploading segments... Dec 28 01:12:07.824150890 (537h43m7.695233935s) [upload=471584]: Upload finished. err: <nil> Duration: 30.002886898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX....X...] duration: 6.031181532s trace: https://jaeger.dev.storj.io/trace/127f0d97861aa614 FAIL (2025-12-28 01:11:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A) context deadline exceeded ======LOGS====== segments... Dec 28 01:09:28.801193842 (537h40m28.672276897s) [upload=471548]: Got next segment. Inline: false Dec 28 01:09:28.801233106 (537h40m28.672316152s) [upload=471548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:09:28.801238136 (537h40m28.672321181s) [upload=471548]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:09:29.042897500 (537h40m28.913980545s) [seg_pos={0 0}, upload=471548]: Begin upload segment... Dec 28 01:09:29.042947525 (537h40m28.91403056s) [seg_pos={0 0}, upload=471548]: Done begin upload segment. Dec 28 01:09:29.042972984 (537h40m28.914056019s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.043107540 (537h40m28.914190585s) [noise=true , address=145.79.227.5:4223 , piece_id=62YJ6M73ZMINB3OIFYN7W4FOUT5VVWNCRZLWJD5VVSPIJQAHNT7Q, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.043664961 (537h40m28.914747996s) [noise=true , address=77.90.31.13:4627 , piece_id=W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044078930 (537h40m28.915161965s) [noise=true , address=77.90.31.17:4770 , piece_id=BC6UHURZ44KJX2RFQQN6EMM4HHI5WV4IG67X7SDGBVLJ3I7ETL7A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044491656 (537h40m28.915574691s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044903099 (537h40m28.915986134s) [noise=true , address=145.79.227.37:4567 , piece_id=GCRXI3NS2PDQB26FJEBPHGL3EZHYHZYMGU5PHEI2M73UWR3S2PLA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.045311987 (537h40m28.916395032s) [noise=true , address=45.91.116.5:4299 , piece_id=6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.045675590 (537h40m28.916758625s) [noise=true , address=145.79.227.13:4804 , piece_id=26WSYCQG4V37QT27AQH4GD3BZDX4TV6MUBXURB3AMEFOG7CK3JTQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.046167026 (537h40m28.917250061s) [noise=true , address=145.79.227.14:4839 , piece_id=AG2HETJMY2WKON2YWI37YPCBXN3GCOD64MNU3YBQ7RG2RMQECWEA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.047361510 (537h40m28.918444565s) [noise=true , address=145.79.227.51:4411 , piece_id=RZ5DZACFISPIW4AWK3IVX6EKTQ7Z3TXJRRQNDPN3MS364VSFW45Q, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.047453355 (537h40m28.91853639s) [noise=true , address=145.79.227.4:4187 , piece_id=566IXCIPWJ2YRRF47YPCI2KMF5MXAPZUZCPI777625DIRPWI3XKA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.048679510 (537h40m28.919762545s) [noise=true , address=145.79.227.9:4367 , piece_id=OK6D3GYMJSWUZ4LOUD5UWN3X525UNSW46U6TCXKREEGB5A27YBLA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.048882977 (537h40m28.919966012s) [noise=true , address=45.91.116.7:4012 , piece_id=5RTNYKRFUUMF6RC342WTRVARAS4CXVNWLEII3SOHDZXIRSSFZ2FQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.052583283 (537h40m28.923666318s) [noise=true , address=145.79.227.17:4970 , piece_id=6ZMSNU4C7QIU3LC45WU3I3U5BCSJMMSTTB4AK7CMHCZCS3Z5XCRA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.055501199 (537h40m28.926584234s) [noise=true , address=145.79.227.38:4979 , piece_id=BHLXMCNPYLK5EJXDDE4QNN4FVPF3XSOMJTGABBTWM6P5YMKZTS3A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.057908001 (537h40m28.928991036s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:30.778836695 (537h40m30.64991973s) [upload=471548]: Next returned nil segment Dec 28 01:09:30.778849680 (537h40m30.649932715s) [upload=471548]: Waiting for error group managing segments... Dec 28 01:09:31.077670747 (537h40m30.948753802s) [noise=true , address=145.79.227.38:4979 , piece_id=BHLXMCNPYLK5EJXDDE4QNN4FVPF3XSOMJTGABBTWM6P5YMKZTS3A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.088961126 (537h40m30.960044171s) [noise=true , address=145.79.227.9:4367 , piece_id=OK6D3GYMJSWUZ4LOUD5UWN3X525UNSW46U6TCXKREEGB5A27YBLA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.117624864 (537h40m30.988707899s) [noise=true , address=145.79.227.5:4223 , piece_id=62YJ6M73ZMINB3OIFYN7W4FOUT5VVWNCRZLWJD5VVSPIJQAHNT7Q, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.154546446 (537h40m31.025629492s) [noise=true , address=145.79.227.17:4970 , piece_id=6ZMSNU4C7QIU3LC45WU3I3U5BCSJMMSTTB4AK7CMHCZCS3Z5XCRA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.159426348 (537h40m31.030509384s) [noise=true , address=145.79.227.51:4411 , piece_id=RZ5DZACFISPIW4AWK3IVX6EKTQ7Z3TXJRRQNDPN3MS364VSFW45Q, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.169486405 (537h40m31.04056944s) [noise=true , address=145.79.227.4:4187 , piece_id=566IXCIPWJ2YRRF47YPCI2KMF5MXAPZUZCPI777625DIRPWI3XKA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.180586650 (537h40m31.051669706s) [noise=true , address=145.79.227.13:4804 , piece_id=26WSYCQG4V37QT27AQH4GD3BZDX4TV6MUBXURB3AMEFOG7CK3JTQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.185086549 (537h40m31.056169594s) [noise=true , address=145.79.227.14:4839 , piece_id=AG2HETJMY2WKON2YWI37YPCBXN3GCOD64MNU3YBQ7RG2RMQECWEA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.318626112 (537h40m31.189709157s) [noise=true , address=77.90.31.13:4627 , piece_id=W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.327514727 (537h40m31.198597762s) [noise=true , address=145.79.227.37:4567 , piece_id=GCRXI3NS2PDQB26FJEBPHGL3EZHYHZYMGU5PHEI2M73UWR3S2PLA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.382328739 (537h40m31.253411774s) [noise=true , address=77.90.31.17:4770 , piece_id=BC6UHURZ44KJX2RFQQN6EMM4HHI5WV4IG67X7SDGBVLJ3I7ETL7A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.404456462 (537h40m31.275539507s) [noise=true , address=45.91.116.5:4299 , piece_id=6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.417026157 (537h40m31.288109202s) [noise=true , address=45.91.116.7:4012 , piece_id=5RTNYKRFUUMF6RC342WTRVARAS4CXVNWLEII3SOHDZXIRSSFZ2FQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.417050112 (537h40m31.288133157s) [seg_pos={0 0}, upload=471548]: Segment reached optimal threshold of 13 pieces. Dec 28 01:09:31.417218453 (537h40m31.288301498s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection Dec 28 01:09:31.417225416 (537h40m31.288308451s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417279199 (537h40m31.288362244s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 01:09:31.417287003 (537h40m31.288370038s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417392634 (537h40m31.288475669s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection Dec 28 01:09:31.417405468 (537h40m31.288488503s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417419215 (537h40m31.28850226s) [seg_pos={0 0}, upload=471548]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:09:31.417494118 (537h40m31.288577163s) [upload=471548]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:09:31.417500299 (537h40m31.288583334s) [upload=471548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:09:31.709439023 (537h40m31.580522058s) [upload=471548]: Done uploading segments... Dec 28 01:09:31.709457898 (537h40m31.580540933s) [upload=471548]: Upload finished. err: <nil> Duration: 6.031209934s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX....X..] duration: 6.1865583s trace: https://jaeger.dev.storj.io/trace/127f0d97861aa614 FAIL (2025-12-28 01:11:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A) context deadline exceeded ======LOGS====== segments... Dec 28 01:09:28.801193842 (537h40m28.672276897s) [upload=471548]: Got next segment. Inline: false Dec 28 01:09:28.801233106 (537h40m28.672316152s) [upload=471548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:09:28.801238136 (537h40m28.672321181s) [upload=471548]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:09:29.042897500 (537h40m28.913980545s) [seg_pos={0 0}, upload=471548]: Begin upload segment... Dec 28 01:09:29.042947525 (537h40m28.91403056s) [seg_pos={0 0}, upload=471548]: Done begin upload segment. Dec 28 01:09:29.042972984 (537h40m28.914056019s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.043107540 (537h40m28.914190585s) [noise=true , address=145.79.227.5:4223 , piece_id=62YJ6M73ZMINB3OIFYN7W4FOUT5VVWNCRZLWJD5VVSPIJQAHNT7Q, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.043664961 (537h40m28.914747996s) [noise=true , address=77.90.31.13:4627 , piece_id=W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044078930 (537h40m28.915161965s) [noise=true , address=77.90.31.17:4770 , piece_id=BC6UHURZ44KJX2RFQQN6EMM4HHI5WV4IG67X7SDGBVLJ3I7ETL7A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044491656 (537h40m28.915574691s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044903099 (537h40m28.915986134s) [noise=true , address=145.79.227.37:4567 , piece_id=GCRXI3NS2PDQB26FJEBPHGL3EZHYHZYMGU5PHEI2M73UWR3S2PLA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.045311987 (537h40m28.916395032s) [noise=true , address=45.91.116.5:4299 , piece_id=6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.045675590 (537h40m28.916758625s) [noise=true , address=145.79.227.13:4804 , piece_id=26WSYCQG4V37QT27AQH4GD3BZDX4TV6MUBXURB3AMEFOG7CK3JTQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.046167026 (537h40m28.917250061s) [noise=true , address=145.79.227.14:4839 , piece_id=AG2HETJMY2WKON2YWI37YPCBXN3GCOD64MNU3YBQ7RG2RMQECWEA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.047361510 (537h40m28.918444565s) [noise=true , address=145.79.227.51:4411 , piece_id=RZ5DZACFISPIW4AWK3IVX6EKTQ7Z3TXJRRQNDPN3MS364VSFW45Q, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.047453355 (537h40m28.91853639s) [noise=true , address=145.79.227.4:4187 , piece_id=566IXCIPWJ2YRRF47YPCI2KMF5MXAPZUZCPI777625DIRPWI3XKA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.048679510 (537h40m28.919762545s) [noise=true , address=145.79.227.9:4367 , piece_id=OK6D3GYMJSWUZ4LOUD5UWN3X525UNSW46U6TCXKREEGB5A27YBLA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.048882977 (537h40m28.919966012s) [noise=true , address=45.91.116.7:4012 , piece_id=5RTNYKRFUUMF6RC342WTRVARAS4CXVNWLEII3SOHDZXIRSSFZ2FQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.052583283 (537h40m28.923666318s) [noise=true , address=145.79.227.17:4970 , piece_id=6ZMSNU4C7QIU3LC45WU3I3U5BCSJMMSTTB4AK7CMHCZCS3Z5XCRA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.055501199 (537h40m28.926584234s) [noise=true , address=145.79.227.38:4979 , piece_id=BHLXMCNPYLK5EJXDDE4QNN4FVPF3XSOMJTGABBTWM6P5YMKZTS3A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.057908001 (537h40m28.928991036s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:30.778836695 (537h40m30.64991973s) [upload=471548]: Next returned nil segment Dec 28 01:09:30.778849680 (537h40m30.649932715s) [upload=471548]: Waiting for error group managing segments... Dec 28 01:09:31.077670747 (537h40m30.948753802s) [noise=true , address=145.79.227.38:4979 , piece_id=BHLXMCNPYLK5EJXDDE4QNN4FVPF3XSOMJTGABBTWM6P5YMKZTS3A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.088961126 (537h40m30.960044171s) [noise=true , address=145.79.227.9:4367 , piece_id=OK6D3GYMJSWUZ4LOUD5UWN3X525UNSW46U6TCXKREEGB5A27YBLA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.117624864 (537h40m30.988707899s) [noise=true , address=145.79.227.5:4223 , piece_id=62YJ6M73ZMINB3OIFYN7W4FOUT5VVWNCRZLWJD5VVSPIJQAHNT7Q, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.154546446 (537h40m31.025629492s) [noise=true , address=145.79.227.17:4970 , piece_id=6ZMSNU4C7QIU3LC45WU3I3U5BCSJMMSTTB4AK7CMHCZCS3Z5XCRA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.159426348 (537h40m31.030509384s) [noise=true , address=145.79.227.51:4411 , piece_id=RZ5DZACFISPIW4AWK3IVX6EKTQ7Z3TXJRRQNDPN3MS364VSFW45Q, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.169486405 (537h40m31.04056944s) [noise=true , address=145.79.227.4:4187 , piece_id=566IXCIPWJ2YRRF47YPCI2KMF5MXAPZUZCPI777625DIRPWI3XKA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.180586650 (537h40m31.051669706s) [noise=true , address=145.79.227.13:4804 , piece_id=26WSYCQG4V37QT27AQH4GD3BZDX4TV6MUBXURB3AMEFOG7CK3JTQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.185086549 (537h40m31.056169594s) [noise=true , address=145.79.227.14:4839 , piece_id=AG2HETJMY2WKON2YWI37YPCBXN3GCOD64MNU3YBQ7RG2RMQECWEA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.318626112 (537h40m31.189709157s) [noise=true , address=77.90.31.13:4627 , piece_id=W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.327514727 (537h40m31.198597762s) [noise=true , address=145.79.227.37:4567 , piece_id=GCRXI3NS2PDQB26FJEBPHGL3EZHYHZYMGU5PHEI2M73UWR3S2PLA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.382328739 (537h40m31.253411774s) [noise=true , address=77.90.31.17:4770 , piece_id=BC6UHURZ44KJX2RFQQN6EMM4HHI5WV4IG67X7SDGBVLJ3I7ETL7A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.404456462 (537h40m31.275539507s) [noise=true , address=45.91.116.5:4299 , piece_id=6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.417026157 (537h40m31.288109202s) [noise=true , address=45.91.116.7:4012 , piece_id=5RTNYKRFUUMF6RC342WTRVARAS4CXVNWLEII3SOHDZXIRSSFZ2FQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.417050112 (537h40m31.288133157s) [seg_pos={0 0}, upload=471548]: Segment reached optimal threshold of 13 pieces. Dec 28 01:09:31.417218453 (537h40m31.288301498s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection Dec 28 01:09:31.417225416 (537h40m31.288308451s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417279199 (537h40m31.288362244s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 01:09:31.417287003 (537h40m31.288370038s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417392634 (537h40m31.288475669s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection Dec 28 01:09:31.417405468 (537h40m31.288488503s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417419215 (537h40m31.28850226s) [seg_pos={0 0}, upload=471548]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:09:31.417494118 (537h40m31.288577163s) [upload=471548]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:09:31.417500299 (537h40m31.288583334s) [upload=471548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:09:31.709439023 (537h40m31.580522058s) [upload=471548]: Done uploading segments... Dec 28 01:09:31.709457898 (537h40m31.580540933s) [upload=471548]: Upload finished. err: <nil> Duration: 6.186587316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX....X.] duration: 10.533847833s trace: https://jaeger.dev.storj.io/trace/127f0d97861aa614 FAIL (2025-12-28 01:10:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A) context deadline exceeded ======LOGS====== segments... Dec 28 01:09:28.801193842 (537h40m28.672276897s) [upload=471548]: Got next segment. Inline: false Dec 28 01:09:28.801233106 (537h40m28.672316152s) [upload=471548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:09:28.801238136 (537h40m28.672321181s) [upload=471548]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:09:29.042897500 (537h40m28.913980545s) [seg_pos={0 0}, upload=471548]: Begin upload segment... Dec 28 01:09:29.042947525 (537h40m28.91403056s) [seg_pos={0 0}, upload=471548]: Done begin upload segment. Dec 28 01:09:29.042972984 (537h40m28.914056019s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.043107540 (537h40m28.914190585s) [noise=true , address=145.79.227.5:4223 , piece_id=62YJ6M73ZMINB3OIFYN7W4FOUT5VVWNCRZLWJD5VVSPIJQAHNT7Q, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.043664961 (537h40m28.914747996s) [noise=true , address=77.90.31.13:4627 , piece_id=W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044078930 (537h40m28.915161965s) [noise=true , address=77.90.31.17:4770 , piece_id=BC6UHURZ44KJX2RFQQN6EMM4HHI5WV4IG67X7SDGBVLJ3I7ETL7A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044491656 (537h40m28.915574691s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044903099 (537h40m28.915986134s) [noise=true , address=145.79.227.37:4567 , piece_id=GCRXI3NS2PDQB26FJEBPHGL3EZHYHZYMGU5PHEI2M73UWR3S2PLA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.045311987 (537h40m28.916395032s) [noise=true , address=45.91.116.5:4299 , piece_id=6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.045675590 (537h40m28.916758625s) [noise=true , address=145.79.227.13:4804 , piece_id=26WSYCQG4V37QT27AQH4GD3BZDX4TV6MUBXURB3AMEFOG7CK3JTQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.046167026 (537h40m28.917250061s) [noise=true , address=145.79.227.14:4839 , piece_id=AG2HETJMY2WKON2YWI37YPCBXN3GCOD64MNU3YBQ7RG2RMQECWEA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.047361510 (537h40m28.918444565s) [noise=true , address=145.79.227.51:4411 , piece_id=RZ5DZACFISPIW4AWK3IVX6EKTQ7Z3TXJRRQNDPN3MS364VSFW45Q, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.047453355 (537h40m28.91853639s) [noise=true , address=145.79.227.4:4187 , piece_id=566IXCIPWJ2YRRF47YPCI2KMF5MXAPZUZCPI777625DIRPWI3XKA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.048679510 (537h40m28.919762545s) [noise=true , address=145.79.227.9:4367 , piece_id=OK6D3GYMJSWUZ4LOUD5UWN3X525UNSW46U6TCXKREEGB5A27YBLA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.048882977 (537h40m28.919966012s) [noise=true , address=45.91.116.7:4012 , piece_id=5RTNYKRFUUMF6RC342WTRVARAS4CXVNWLEII3SOHDZXIRSSFZ2FQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.052583283 (537h40m28.923666318s) [noise=true , address=145.79.227.17:4970 , piece_id=6ZMSNU4C7QIU3LC45WU3I3U5BCSJMMSTTB4AK7CMHCZCS3Z5XCRA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.055501199 (537h40m28.926584234s) [noise=true , address=145.79.227.38:4979 , piece_id=BHLXMCNPYLK5EJXDDE4QNN4FVPF3XSOMJTGABBTWM6P5YMKZTS3A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.057908001 (537h40m28.928991036s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:30.778836695 (537h40m30.64991973s) [upload=471548]: Next returned nil segment Dec 28 01:09:30.778849680 (537h40m30.649932715s) [upload=471548]: Waiting for error group managing segments... Dec 28 01:09:31.077670747 (537h40m30.948753802s) [noise=true , address=145.79.227.38:4979 , piece_id=BHLXMCNPYLK5EJXDDE4QNN4FVPF3XSOMJTGABBTWM6P5YMKZTS3A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.088961126 (537h40m30.960044171s) [noise=true , address=145.79.227.9:4367 , piece_id=OK6D3GYMJSWUZ4LOUD5UWN3X525UNSW46U6TCXKREEGB5A27YBLA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.117624864 (537h40m30.988707899s) [noise=true , address=145.79.227.5:4223 , piece_id=62YJ6M73ZMINB3OIFYN7W4FOUT5VVWNCRZLWJD5VVSPIJQAHNT7Q, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.154546446 (537h40m31.025629492s) [noise=true , address=145.79.227.17:4970 , piece_id=6ZMSNU4C7QIU3LC45WU3I3U5BCSJMMSTTB4AK7CMHCZCS3Z5XCRA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.159426348 (537h40m31.030509384s) [noise=true , address=145.79.227.51:4411 , piece_id=RZ5DZACFISPIW4AWK3IVX6EKTQ7Z3TXJRRQNDPN3MS364VSFW45Q, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.169486405 (537h40m31.04056944s) [noise=true , address=145.79.227.4:4187 , piece_id=566IXCIPWJ2YRRF47YPCI2KMF5MXAPZUZCPI777625DIRPWI3XKA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.180586650 (537h40m31.051669706s) [noise=true , address=145.79.227.13:4804 , piece_id=26WSYCQG4V37QT27AQH4GD3BZDX4TV6MUBXURB3AMEFOG7CK3JTQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.185086549 (537h40m31.056169594s) [noise=true , address=145.79.227.14:4839 , piece_id=AG2HETJMY2WKON2YWI37YPCBXN3GCOD64MNU3YBQ7RG2RMQECWEA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.318626112 (537h40m31.189709157s) [noise=true , address=77.90.31.13:4627 , piece_id=W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.327514727 (537h40m31.198597762s) [noise=true , address=145.79.227.37:4567 , piece_id=GCRXI3NS2PDQB26FJEBPHGL3EZHYHZYMGU5PHEI2M73UWR3S2PLA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.382328739 (537h40m31.253411774s) [noise=true , address=77.90.31.17:4770 , piece_id=BC6UHURZ44KJX2RFQQN6EMM4HHI5WV4IG67X7SDGBVLJ3I7ETL7A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.404456462 (537h40m31.275539507s) [noise=true , address=45.91.116.5:4299 , piece_id=6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.417026157 (537h40m31.288109202s) [noise=true , address=45.91.116.7:4012 , piece_id=5RTNYKRFUUMF6RC342WTRVARAS4CXVNWLEII3SOHDZXIRSSFZ2FQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.417050112 (537h40m31.288133157s) [seg_pos={0 0}, upload=471548]: Segment reached optimal threshold of 13 pieces. Dec 28 01:09:31.417218453 (537h40m31.288301498s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection Dec 28 01:09:31.417225416 (537h40m31.288308451s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417279199 (537h40m31.288362244s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 01:09:31.417287003 (537h40m31.288370038s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417392634 (537h40m31.288475669s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection Dec 28 01:09:31.417405468 (537h40m31.288488503s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417419215 (537h40m31.28850226s) [seg_pos={0 0}, upload=471548]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:09:31.417494118 (537h40m31.288577163s) [upload=471548]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:09:31.417500299 (537h40m31.288583334s) [upload=471548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:09:31.709439023 (537h40m31.580522058s) [upload=471548]: Done uploading segments... Dec 28 01:09:31.709457898 (537h40m31.580540933s) [upload=471548]: Upload finished. err: <nil> Duration: 10.533874314s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX....X] duration: 30.003077003s trace: https://jaeger.dev.storj.io/trace/127f0d97861aa614 FAIL (2025-12-28 01:09:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A) context deadline exceeded ======LOGS====== segments... Dec 28 01:09:28.801193842 (537h40m28.672276897s) [upload=471548]: Got next segment. Inline: false Dec 28 01:09:28.801233106 (537h40m28.672316152s) [upload=471548]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:09:28.801238136 (537h40m28.672321181s) [upload=471548]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:09:29.042897500 (537h40m28.913980545s) [seg_pos={0 0}, upload=471548]: Begin upload segment... Dec 28 01:09:29.042947525 (537h40m28.91403056s) [seg_pos={0 0}, upload=471548]: Done begin upload segment. Dec 28 01:09:29.042972984 (537h40m28.914056019s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.043107540 (537h40m28.914190585s) [noise=true , address=145.79.227.5:4223 , piece_id=62YJ6M73ZMINB3OIFYN7W4FOUT5VVWNCRZLWJD5VVSPIJQAHNT7Q, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.043664961 (537h40m28.914747996s) [noise=true , address=77.90.31.13:4627 , piece_id=W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044078930 (537h40m28.915161965s) [noise=true , address=77.90.31.17:4770 , piece_id=BC6UHURZ44KJX2RFQQN6EMM4HHI5WV4IG67X7SDGBVLJ3I7ETL7A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044491656 (537h40m28.915574691s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.044903099 (537h40m28.915986134s) [noise=true , address=145.79.227.37:4567 , piece_id=GCRXI3NS2PDQB26FJEBPHGL3EZHYHZYMGU5PHEI2M73UWR3S2PLA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.045311987 (537h40m28.916395032s) [noise=true , address=45.91.116.5:4299 , piece_id=6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.045675590 (537h40m28.916758625s) [noise=true , address=145.79.227.13:4804 , piece_id=26WSYCQG4V37QT27AQH4GD3BZDX4TV6MUBXURB3AMEFOG7CK3JTQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.046167026 (537h40m28.917250061s) [noise=true , address=145.79.227.14:4839 , piece_id=AG2HETJMY2WKON2YWI37YPCBXN3GCOD64MNU3YBQ7RG2RMQECWEA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.047361510 (537h40m28.918444565s) [noise=true , address=145.79.227.51:4411 , piece_id=RZ5DZACFISPIW4AWK3IVX6EKTQ7Z3TXJRRQNDPN3MS364VSFW45Q, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.047453355 (537h40m28.91853639s) [noise=true , address=145.79.227.4:4187 , piece_id=566IXCIPWJ2YRRF47YPCI2KMF5MXAPZUZCPI777625DIRPWI3XKA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.048679510 (537h40m28.919762545s) [noise=true , address=145.79.227.9:4367 , piece_id=OK6D3GYMJSWUZ4LOUD5UWN3X525UNSW46U6TCXKREEGB5A27YBLA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.048882977 (537h40m28.919966012s) [noise=true , address=45.91.116.7:4012 , piece_id=5RTNYKRFUUMF6RC342WTRVARAS4CXVNWLEII3SOHDZXIRSSFZ2FQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.052583283 (537h40m28.923666318s) [noise=true , address=145.79.227.17:4970 , piece_id=6ZMSNU4C7QIU3LC45WU3I3U5BCSJMMSTTB4AK7CMHCZCS3Z5XCRA, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.055501199 (537h40m28.926584234s) [noise=true , address=145.79.227.38:4979 , piece_id=BHLXMCNPYLK5EJXDDE4QNN4FVPF3XSOMJTGABBTWM6P5YMKZTS3A, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:29.057908001 (537h40m28.928991036s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Uploading piece... Dec 28 01:09:30.778836695 (537h40m30.64991973s) [upload=471548]: Next returned nil segment Dec 28 01:09:30.778849680 (537h40m30.649932715s) [upload=471548]: Waiting for error group managing segments... Dec 28 01:09:31.077670747 (537h40m30.948753802s) [noise=true , address=145.79.227.38:4979 , piece_id=BHLXMCNPYLK5EJXDDE4QNN4FVPF3XSOMJTGABBTWM6P5YMKZTS3A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.088961126 (537h40m30.960044171s) [noise=true , address=145.79.227.9:4367 , piece_id=OK6D3GYMJSWUZ4LOUD5UWN3X525UNSW46U6TCXKREEGB5A27YBLA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.117624864 (537h40m30.988707899s) [noise=true , address=145.79.227.5:4223 , piece_id=62YJ6M73ZMINB3OIFYN7W4FOUT5VVWNCRZLWJD5VVSPIJQAHNT7Q, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.154546446 (537h40m31.025629492s) [noise=true , address=145.79.227.17:4970 , piece_id=6ZMSNU4C7QIU3LC45WU3I3U5BCSJMMSTTB4AK7CMHCZCS3Z5XCRA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.159426348 (537h40m31.030509384s) [noise=true , address=145.79.227.51:4411 , piece_id=RZ5DZACFISPIW4AWK3IVX6EKTQ7Z3TXJRRQNDPN3MS364VSFW45Q, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.169486405 (537h40m31.04056944s) [noise=true , address=145.79.227.4:4187 , piece_id=566IXCIPWJ2YRRF47YPCI2KMF5MXAPZUZCPI777625DIRPWI3XKA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.180586650 (537h40m31.051669706s) [noise=true , address=145.79.227.13:4804 , piece_id=26WSYCQG4V37QT27AQH4GD3BZDX4TV6MUBXURB3AMEFOG7CK3JTQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.185086549 (537h40m31.056169594s) [noise=true , address=145.79.227.14:4839 , piece_id=AG2HETJMY2WKON2YWI37YPCBXN3GCOD64MNU3YBQ7RG2RMQECWEA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.318626112 (537h40m31.189709157s) [noise=true , address=77.90.31.13:4627 , piece_id=W3HROFBWV36AZME6G4EOSRQUHGDMNTLBJZL4LFAMT2XPUT3YGJUA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.327514727 (537h40m31.198597762s) [noise=true , address=145.79.227.37:4567 , piece_id=GCRXI3NS2PDQB26FJEBPHGL3EZHYHZYMGU5PHEI2M73UWR3S2PLA, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.382328739 (537h40m31.253411774s) [noise=true , address=77.90.31.17:4770 , piece_id=BC6UHURZ44KJX2RFQQN6EMM4HHI5WV4IG67X7SDGBVLJ3I7ETL7A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.404456462 (537h40m31.275539507s) [noise=true , address=45.91.116.5:4299 , piece_id=6MLATZRUAXWKEPBQ5UXQB47MF7U4OQTBOXYQZVD6V2URMSV4HK6A, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.417026157 (537h40m31.288109202s) [noise=true , address=45.91.116.7:4012 , piece_id=5RTNYKRFUUMF6RC342WTRVARAS4CXVNWLEII3SOHDZXIRSSFZ2FQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: <nil> Dec 28 01:09:31.417050112 (537h40m31.288133157s) [seg_pos={0 0}, upload=471548]: Segment reached optimal threshold of 13 pieces. Dec 28 01:09:31.417218453 (537h40m31.288301498s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42310->45.91.116.4:4939: use of closed network connection Dec 28 01:09:31.417225416 (537h40m31.288308451s) [noise=true , address=45.91.116.4:4939 , piece_id=CHWDULNFMXVH3YFFBI7AGYPPHTUKO4BCP5J6X3ABWZAIWQ5FVWOQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417279199 (537h40m31.288362244s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 01:09:31.417287003 (537h40m31.288370038s) [noise=true , address=45.91.116.10:4084 , piece_id=OQEVYPYLHFXXJOGHPJFHAA62A7V75S5LEXY7PHTHEGUKCN25ZMQQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417392634 (537h40m31.288475669s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37928->45.91.116.6:4974: use of closed network connection Dec 28 01:09:31.417405468 (537h40m31.288488503s) [noise=true , address=45.91.116.6:4974 , piece_id=ZRODJGNJHLYAA5U5IAG2HYRYX3OQTIIK4GTWW7GOI7GOGRP33YLQ, seg_pos={0 0}, upload=471548]: Long tail cancellation due to optimal threshold Dec 28 01:09:31.417419215 (537h40m31.28850226s) [seg_pos={0 0}, upload=471548]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:09:31.417494118 (537h40m31.288577163s) [upload=471548]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:09:31.417500299 (537h40m31.288583334s) [upload=471548]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:09:31.709439023 (537h40m31.580522058s) [upload=471548]: Done uploading segments... Dec 28 01:09:31.709457898 (537h40m31.580540933s) [upload=471548]: Upload finished. err: <nil> Duration: 30.003103394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX....] duration: 16.949457387s trace: https://jaeger.dev.storj.io/trace/30a26f0522322093 FAIL (2025-12-28 01:09:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA) context deadline exceeded ======LOGS====== :06:29.553789671 (537h37m29.424872716s) [upload=471506]: Starting upload Dec 28 01:06:29.553880263 (537h37m29.424963298s) [upload=471506]: Done starting upload Dec 28 01:06:29.553926091 (537h37m29.425009126s) [upload=471506]: Uploading segments... Dec 28 01:06:29.554081627 (537h37m29.425164662s) [upload=471506]: Got next segment. Inline: false Dec 28 01:06:29.554126372 (537h37m29.425209407s) [upload=471506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:06:29.554132474 (537h37m29.425215519s) [upload=471506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:06:29.798186648 (537h37m29.669269683s) [seg_pos={0 0}, upload=471506]: Begin upload segment... Dec 28 01:06:29.798237866 (537h37m29.669320941s) [seg_pos={0 0}, upload=471506]: Done begin upload segment. Dec 28 01:06:29.798292840 (537h37m29.669375875s) [noise=true , address=145.79.227.11:4731 , piece_id=YHJ5KGPWPQVLD4RVJXFMMNODXHCYX2CCPCGQDF2WASQSVHDFSVEQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.798304803 (537h37m29.669387838s) [noise=true , address=77.90.31.10:4517 , piece_id=AZCYGK7ZSALKHNV3THLF56AM5RMORCTJUCA2YOACYOEEBPQZUQ4A, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.798805466 (537h37m29.669888511s) [noise=true , address=145.79.227.46:4603 , piece_id=KV3WWPOSNPOXREXWKK22P2S3MURAOIU6UDSAJZU5YD5VDDQ7LS2Q, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.799191601 (537h37m29.670274636s) [noise=true , address=145.79.227.39:4014 , piece_id=PDH3T2IIT47HD2ELF3IOB4UGBPVPM66KE2EUJIHKAMGU2WS3DIZA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.799604598 (537h37m29.670687633s) [noise=true , address=77.90.31.13:4627 , piece_id=COLW6V55W7BAH3QKDPFKGRNMKEPUNXJZEOO337V246Q2CH7JAP4A, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800069864 (537h37m29.6711529s) [noise=true , address=145.79.227.45:4231 , piece_id=MC4QFPM7CX56ASQBYWMARUJCM77DE77A26LKXZXJDCKQZHDS7BAA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800540660 (537h37m29.671623695s) [noise=true , address=45.91.116.4:4939 , piece_id=CVRDL5CUWHVEH6XH4H3DIFZBFSE6D3IDSIYYPIRXD5P2B6XK4IEQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800948367 (537h37m29.672031412s) [noise=true , address=145.79.227.52:4446 , piece_id=3UXUTOXVHKWNKZJNMO3N3DFP2R6FAIFP24O4I4AXI6WWYO7MIUWA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801347978 (537h37m29.672431013s) [noise=true , address=45.91.116.13:4156 , piece_id=RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801734002 (537h37m29.672817048s) [noise=true , address=145.79.227.47:4266 , piece_id=ZMCJUICWQKQPHHNSNJLJB3D6YSOUCM6VZTNNTJTFDXAVHYS3S6TQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801898556 (537h37m29.672981601s) [noise=true , address=77.90.31.11:4556 , piece_id=VBR5YYAC7SF7RKO67CTRMOQ5ZT3TPE7ZUFXNDUXXBSZIPVIXHLQA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.802371667 (537h37m29.673454712s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.802698579 (537h37m29.673781625s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.803122547 (537h37m29.674205592s) [noise=true , address=77.90.31.14:4663 , piece_id=NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.803604314 (537h37m29.674687349s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.809721563 (537h37m29.680804598s) [noise=true , address=45.91.116.6:4975 , piece_id=PUMZ6TTLVNWBKD66TW7KSIFMOOGMGQ7ZJ54BQVGTPNFZDFLMGVQA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:31.533662902 (537h37m31.404745957s) [upload=471506]: Next returned nil segment Dec 28 01:06:31.533679524 (537h37m31.404762559s) [upload=471506]: Waiting for error group managing segments... Dec 28 01:06:31.828015661 (537h37m31.699098706s) [noise=true , address=145.79.227.46:4603 , piece_id=KV3WWPOSNPOXREXWKK22P2S3MURAOIU6UDSAJZU5YD5VDDQ7LS2Q, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.833605936 (537h37m31.704688981s) [noise=true , address=145.79.227.39:4014 , piece_id=PDH3T2IIT47HD2ELF3IOB4UGBPVPM66KE2EUJIHKAMGU2WS3DIZA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.836764400 (537h37m31.707847445s) [noise=true , address=145.79.227.11:4731 , piece_id=YHJ5KGPWPQVLD4RVJXFMMNODXHCYX2CCPCGQDF2WASQSVHDFSVEQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.840430000 (537h37m31.711513035s) [noise=true , address=145.79.227.45:4231 , piece_id=MC4QFPM7CX56ASQBYWMARUJCM77DE77A26LKXZXJDCKQZHDS7BAA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.840442403 (537h37m31.711525448s) [noise=true , address=145.79.227.47:4266 , piece_id=ZMCJUICWQKQPHHNSNJLJB3D6YSOUCM6VZTNNTJTFDXAVHYS3S6TQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.869641091 (537h37m31.740724126s) [noise=true , address=145.79.227.52:4446 , piece_id=3UXUTOXVHKWNKZJNMO3N3DFP2R6FAIFP24O4I4AXI6WWYO7MIUWA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.025232312 (537h37m31.896315358s) [noise=true , address=77.90.31.11:4556 , piece_id=VBR5YYAC7SF7RKO67CTRMOQ5ZT3TPE7ZUFXNDUXXBSZIPVIXHLQA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.025868604 (537h37m31.896951649s) [noise=true , address=77.90.31.13:4627 , piece_id=COLW6V55W7BAH3QKDPFKGRNMKEPUNXJZEOO337V246Q2CH7JAP4A, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.059586988 (537h37m31.930670033s) [noise=true , address=77.90.31.14:4663 , piece_id=NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.074009770 (537h37m31.945092815s) [noise=true , address=45.91.116.6:4975 , piece_id=PUMZ6TTLVNWBKD66TW7KSIFMOOGMGQ7ZJ54BQVGTPNFZDFLMGVQA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.076623106 (537h37m31.947706141s) [noise=true , address=77.90.31.10:4517 , piece_id=AZCYGK7ZSALKHNV3THLF56AM5RMORCTJUCA2YOACYOEEBPQZUQ4A, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.111222067 (537h37m31.982305102s) [noise=true , address=45.91.116.4:4939 , piece_id=CVRDL5CUWHVEH6XH4H3DIFZBFSE6D3IDSIYYPIRXD5P2B6XK4IEQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.193481548 (537h37m32.064564583s) [noise=true , address=45.91.116.13:4156 , piece_id=RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.193513679 (537h37m32.064596724s) [seg_pos={0 0}, upload=471506]: Segment reached optimal threshold of 13 pieces. Dec 28 01:06:32.193735521 (537h37m32.064818566s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:32.193742925 (537h37m32.06482597s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194041604 (537h37m32.065124649s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection Dec 28 01:06:32.194049309 (537h37m32.065132344s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194222388 (537h37m32.065305443s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:32.194244140 (537h37m32.065327195s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194278274 (537h37m32.065361319s) [seg_pos={0 0}, upload=471506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:06:32.194499506 (537h37m32.065582551s) [upload=471506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:06:32.194505918 (537h37m32.065588963s) [upload=471506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:06:32.496398003 (537h37m32.367481038s) [upload=471506]: Done uploading segments... Dec 28 01:06:32.496415877 (537h37m32.367498922s) [upload=471506]: Upload finished. err: <nil> Duration: 16.94948518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX...] duration: 17.983410618s trace: https://jaeger.dev.storj.io/trace/30a26f0522322093 FAIL (2025-12-28 01:08:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA) context deadline exceeded ======LOGS====== :06:29.553789671 (537h37m29.424872716s) [upload=471506]: Starting upload Dec 28 01:06:29.553880263 (537h37m29.424963298s) [upload=471506]: Done starting upload Dec 28 01:06:29.553926091 (537h37m29.425009126s) [upload=471506]: Uploading segments... Dec 28 01:06:29.554081627 (537h37m29.425164662s) [upload=471506]: Got next segment. Inline: false Dec 28 01:06:29.554126372 (537h37m29.425209407s) [upload=471506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:06:29.554132474 (537h37m29.425215519s) [upload=471506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:06:29.798186648 (537h37m29.669269683s) [seg_pos={0 0}, upload=471506]: Begin upload segment... Dec 28 01:06:29.798237866 (537h37m29.669320941s) [seg_pos={0 0}, upload=471506]: Done begin upload segment. Dec 28 01:06:29.798292840 (537h37m29.669375875s) [noise=true , address=145.79.227.11:4731 , piece_id=YHJ5KGPWPQVLD4RVJXFMMNODXHCYX2CCPCGQDF2WASQSVHDFSVEQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.798304803 (537h37m29.669387838s) [noise=true , address=77.90.31.10:4517 , piece_id=AZCYGK7ZSALKHNV3THLF56AM5RMORCTJUCA2YOACYOEEBPQZUQ4A, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.798805466 (537h37m29.669888511s) [noise=true , address=145.79.227.46:4603 , piece_id=KV3WWPOSNPOXREXWKK22P2S3MURAOIU6UDSAJZU5YD5VDDQ7LS2Q, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.799191601 (537h37m29.670274636s) [noise=true , address=145.79.227.39:4014 , piece_id=PDH3T2IIT47HD2ELF3IOB4UGBPVPM66KE2EUJIHKAMGU2WS3DIZA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.799604598 (537h37m29.670687633s) [noise=true , address=77.90.31.13:4627 , piece_id=COLW6V55W7BAH3QKDPFKGRNMKEPUNXJZEOO337V246Q2CH7JAP4A, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800069864 (537h37m29.6711529s) [noise=true , address=145.79.227.45:4231 , piece_id=MC4QFPM7CX56ASQBYWMARUJCM77DE77A26LKXZXJDCKQZHDS7BAA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800540660 (537h37m29.671623695s) [noise=true , address=45.91.116.4:4939 , piece_id=CVRDL5CUWHVEH6XH4H3DIFZBFSE6D3IDSIYYPIRXD5P2B6XK4IEQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800948367 (537h37m29.672031412s) [noise=true , address=145.79.227.52:4446 , piece_id=3UXUTOXVHKWNKZJNMO3N3DFP2R6FAIFP24O4I4AXI6WWYO7MIUWA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801347978 (537h37m29.672431013s) [noise=true , address=45.91.116.13:4156 , piece_id=RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801734002 (537h37m29.672817048s) [noise=true , address=145.79.227.47:4266 , piece_id=ZMCJUICWQKQPHHNSNJLJB3D6YSOUCM6VZTNNTJTFDXAVHYS3S6TQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801898556 (537h37m29.672981601s) [noise=true , address=77.90.31.11:4556 , piece_id=VBR5YYAC7SF7RKO67CTRMOQ5ZT3TPE7ZUFXNDUXXBSZIPVIXHLQA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.802371667 (537h37m29.673454712s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.802698579 (537h37m29.673781625s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.803122547 (537h37m29.674205592s) [noise=true , address=77.90.31.14:4663 , piece_id=NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.803604314 (537h37m29.674687349s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.809721563 (537h37m29.680804598s) [noise=true , address=45.91.116.6:4975 , piece_id=PUMZ6TTLVNWBKD66TW7KSIFMOOGMGQ7ZJ54BQVGTPNFZDFLMGVQA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:31.533662902 (537h37m31.404745957s) [upload=471506]: Next returned nil segment Dec 28 01:06:31.533679524 (537h37m31.404762559s) [upload=471506]: Waiting for error group managing segments... Dec 28 01:06:31.828015661 (537h37m31.699098706s) [noise=true , address=145.79.227.46:4603 , piece_id=KV3WWPOSNPOXREXWKK22P2S3MURAOIU6UDSAJZU5YD5VDDQ7LS2Q, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.833605936 (537h37m31.704688981s) [noise=true , address=145.79.227.39:4014 , piece_id=PDH3T2IIT47HD2ELF3IOB4UGBPVPM66KE2EUJIHKAMGU2WS3DIZA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.836764400 (537h37m31.707847445s) [noise=true , address=145.79.227.11:4731 , piece_id=YHJ5KGPWPQVLD4RVJXFMMNODXHCYX2CCPCGQDF2WASQSVHDFSVEQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.840430000 (537h37m31.711513035s) [noise=true , address=145.79.227.45:4231 , piece_id=MC4QFPM7CX56ASQBYWMARUJCM77DE77A26LKXZXJDCKQZHDS7BAA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.840442403 (537h37m31.711525448s) [noise=true , address=145.79.227.47:4266 , piece_id=ZMCJUICWQKQPHHNSNJLJB3D6YSOUCM6VZTNNTJTFDXAVHYS3S6TQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.869641091 (537h37m31.740724126s) [noise=true , address=145.79.227.52:4446 , piece_id=3UXUTOXVHKWNKZJNMO3N3DFP2R6FAIFP24O4I4AXI6WWYO7MIUWA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.025232312 (537h37m31.896315358s) [noise=true , address=77.90.31.11:4556 , piece_id=VBR5YYAC7SF7RKO67CTRMOQ5ZT3TPE7ZUFXNDUXXBSZIPVIXHLQA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.025868604 (537h37m31.896951649s) [noise=true , address=77.90.31.13:4627 , piece_id=COLW6V55W7BAH3QKDPFKGRNMKEPUNXJZEOO337V246Q2CH7JAP4A, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.059586988 (537h37m31.930670033s) [noise=true , address=77.90.31.14:4663 , piece_id=NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.074009770 (537h37m31.945092815s) [noise=true , address=45.91.116.6:4975 , piece_id=PUMZ6TTLVNWBKD66TW7KSIFMOOGMGQ7ZJ54BQVGTPNFZDFLMGVQA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.076623106 (537h37m31.947706141s) [noise=true , address=77.90.31.10:4517 , piece_id=AZCYGK7ZSALKHNV3THLF56AM5RMORCTJUCA2YOACYOEEBPQZUQ4A, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.111222067 (537h37m31.982305102s) [noise=true , address=45.91.116.4:4939 , piece_id=CVRDL5CUWHVEH6XH4H3DIFZBFSE6D3IDSIYYPIRXD5P2B6XK4IEQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.193481548 (537h37m32.064564583s) [noise=true , address=45.91.116.13:4156 , piece_id=RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.193513679 (537h37m32.064596724s) [seg_pos={0 0}, upload=471506]: Segment reached optimal threshold of 13 pieces. Dec 28 01:06:32.193735521 (537h37m32.064818566s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:32.193742925 (537h37m32.06482597s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194041604 (537h37m32.065124649s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection Dec 28 01:06:32.194049309 (537h37m32.065132344s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194222388 (537h37m32.065305443s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:32.194244140 (537h37m32.065327195s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194278274 (537h37m32.065361319s) [seg_pos={0 0}, upload=471506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:06:32.194499506 (537h37m32.065582551s) [upload=471506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:06:32.194505918 (537h37m32.065588963s) [upload=471506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:06:32.496398003 (537h37m32.367481038s) [upload=471506]: Done uploading segments... Dec 28 01:06:32.496415877 (537h37m32.367498922s) [upload=471506]: Upload finished. err: <nil> Duration: 17.983430465s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX..] duration: 29.90550293s trace: https://jaeger.dev.storj.io/trace/30a26f0522322093 FAIL (2025-12-28 01:08:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA) context deadline exceeded ======LOGS====== :06:29.553789671 (537h37m29.424872716s) [upload=471506]: Starting upload Dec 28 01:06:29.553880263 (537h37m29.424963298s) [upload=471506]: Done starting upload Dec 28 01:06:29.553926091 (537h37m29.425009126s) [upload=471506]: Uploading segments... Dec 28 01:06:29.554081627 (537h37m29.425164662s) [upload=471506]: Got next segment. Inline: false Dec 28 01:06:29.554126372 (537h37m29.425209407s) [upload=471506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:06:29.554132474 (537h37m29.425215519s) [upload=471506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:06:29.798186648 (537h37m29.669269683s) [seg_pos={0 0}, upload=471506]: Begin upload segment... Dec 28 01:06:29.798237866 (537h37m29.669320941s) [seg_pos={0 0}, upload=471506]: Done begin upload segment. Dec 28 01:06:29.798292840 (537h37m29.669375875s) [noise=true , address=145.79.227.11:4731 , piece_id=YHJ5KGPWPQVLD4RVJXFMMNODXHCYX2CCPCGQDF2WASQSVHDFSVEQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.798304803 (537h37m29.669387838s) [noise=true , address=77.90.31.10:4517 , piece_id=AZCYGK7ZSALKHNV3THLF56AM5RMORCTJUCA2YOACYOEEBPQZUQ4A, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.798805466 (537h37m29.669888511s) [noise=true , address=145.79.227.46:4603 , piece_id=KV3WWPOSNPOXREXWKK22P2S3MURAOIU6UDSAJZU5YD5VDDQ7LS2Q, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.799191601 (537h37m29.670274636s) [noise=true , address=145.79.227.39:4014 , piece_id=PDH3T2IIT47HD2ELF3IOB4UGBPVPM66KE2EUJIHKAMGU2WS3DIZA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.799604598 (537h37m29.670687633s) [noise=true , address=77.90.31.13:4627 , piece_id=COLW6V55W7BAH3QKDPFKGRNMKEPUNXJZEOO337V246Q2CH7JAP4A, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800069864 (537h37m29.6711529s) [noise=true , address=145.79.227.45:4231 , piece_id=MC4QFPM7CX56ASQBYWMARUJCM77DE77A26LKXZXJDCKQZHDS7BAA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800540660 (537h37m29.671623695s) [noise=true , address=45.91.116.4:4939 , piece_id=CVRDL5CUWHVEH6XH4H3DIFZBFSE6D3IDSIYYPIRXD5P2B6XK4IEQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800948367 (537h37m29.672031412s) [noise=true , address=145.79.227.52:4446 , piece_id=3UXUTOXVHKWNKZJNMO3N3DFP2R6FAIFP24O4I4AXI6WWYO7MIUWA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801347978 (537h37m29.672431013s) [noise=true , address=45.91.116.13:4156 , piece_id=RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801734002 (537h37m29.672817048s) [noise=true , address=145.79.227.47:4266 , piece_id=ZMCJUICWQKQPHHNSNJLJB3D6YSOUCM6VZTNNTJTFDXAVHYS3S6TQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801898556 (537h37m29.672981601s) [noise=true , address=77.90.31.11:4556 , piece_id=VBR5YYAC7SF7RKO67CTRMOQ5ZT3TPE7ZUFXNDUXXBSZIPVIXHLQA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.802371667 (537h37m29.673454712s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.802698579 (537h37m29.673781625s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.803122547 (537h37m29.674205592s) [noise=true , address=77.90.31.14:4663 , piece_id=NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.803604314 (537h37m29.674687349s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.809721563 (537h37m29.680804598s) [noise=true , address=45.91.116.6:4975 , piece_id=PUMZ6TTLVNWBKD66TW7KSIFMOOGMGQ7ZJ54BQVGTPNFZDFLMGVQA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:31.533662902 (537h37m31.404745957s) [upload=471506]: Next returned nil segment Dec 28 01:06:31.533679524 (537h37m31.404762559s) [upload=471506]: Waiting for error group managing segments... Dec 28 01:06:31.828015661 (537h37m31.699098706s) [noise=true , address=145.79.227.46:4603 , piece_id=KV3WWPOSNPOXREXWKK22P2S3MURAOIU6UDSAJZU5YD5VDDQ7LS2Q, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.833605936 (537h37m31.704688981s) [noise=true , address=145.79.227.39:4014 , piece_id=PDH3T2IIT47HD2ELF3IOB4UGBPVPM66KE2EUJIHKAMGU2WS3DIZA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.836764400 (537h37m31.707847445s) [noise=true , address=145.79.227.11:4731 , piece_id=YHJ5KGPWPQVLD4RVJXFMMNODXHCYX2CCPCGQDF2WASQSVHDFSVEQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.840430000 (537h37m31.711513035s) [noise=true , address=145.79.227.45:4231 , piece_id=MC4QFPM7CX56ASQBYWMARUJCM77DE77A26LKXZXJDCKQZHDS7BAA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.840442403 (537h37m31.711525448s) [noise=true , address=145.79.227.47:4266 , piece_id=ZMCJUICWQKQPHHNSNJLJB3D6YSOUCM6VZTNNTJTFDXAVHYS3S6TQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.869641091 (537h37m31.740724126s) [noise=true , address=145.79.227.52:4446 , piece_id=3UXUTOXVHKWNKZJNMO3N3DFP2R6FAIFP24O4I4AXI6WWYO7MIUWA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.025232312 (537h37m31.896315358s) [noise=true , address=77.90.31.11:4556 , piece_id=VBR5YYAC7SF7RKO67CTRMOQ5ZT3TPE7ZUFXNDUXXBSZIPVIXHLQA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.025868604 (537h37m31.896951649s) [noise=true , address=77.90.31.13:4627 , piece_id=COLW6V55W7BAH3QKDPFKGRNMKEPUNXJZEOO337V246Q2CH7JAP4A, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.059586988 (537h37m31.930670033s) [noise=true , address=77.90.31.14:4663 , piece_id=NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.074009770 (537h37m31.945092815s) [noise=true , address=45.91.116.6:4975 , piece_id=PUMZ6TTLVNWBKD66TW7KSIFMOOGMGQ7ZJ54BQVGTPNFZDFLMGVQA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.076623106 (537h37m31.947706141s) [noise=true , address=77.90.31.10:4517 , piece_id=AZCYGK7ZSALKHNV3THLF56AM5RMORCTJUCA2YOACYOEEBPQZUQ4A, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.111222067 (537h37m31.982305102s) [noise=true , address=45.91.116.4:4939 , piece_id=CVRDL5CUWHVEH6XH4H3DIFZBFSE6D3IDSIYYPIRXD5P2B6XK4IEQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.193481548 (537h37m32.064564583s) [noise=true , address=45.91.116.13:4156 , piece_id=RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.193513679 (537h37m32.064596724s) [seg_pos={0 0}, upload=471506]: Segment reached optimal threshold of 13 pieces. Dec 28 01:06:32.193735521 (537h37m32.064818566s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:32.193742925 (537h37m32.06482597s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194041604 (537h37m32.065124649s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection Dec 28 01:06:32.194049309 (537h37m32.065132344s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194222388 (537h37m32.065305443s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:32.194244140 (537h37m32.065327195s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194278274 (537h37m32.065361319s) [seg_pos={0 0}, upload=471506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:06:32.194499506 (537h37m32.065582551s) [upload=471506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:06:32.194505918 (537h37m32.065588963s) [upload=471506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:06:32.496398003 (537h37m32.367481038s) [upload=471506]: Done uploading segments... Dec 28 01:06:32.496415877 (537h37m32.367498922s) [upload=471506]: Upload finished. err: <nil> Duration: 29.905590807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX.] duration: 18.115283248s trace: https://jaeger.dev.storj.io/trace/30a26f0522322093 FAIL (2025-12-28 01:07:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA) context deadline exceeded ======LOGS====== :06:29.553789671 (537h37m29.424872716s) [upload=471506]: Starting upload Dec 28 01:06:29.553880263 (537h37m29.424963298s) [upload=471506]: Done starting upload Dec 28 01:06:29.553926091 (537h37m29.425009126s) [upload=471506]: Uploading segments... Dec 28 01:06:29.554081627 (537h37m29.425164662s) [upload=471506]: Got next segment. Inline: false Dec 28 01:06:29.554126372 (537h37m29.425209407s) [upload=471506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:06:29.554132474 (537h37m29.425215519s) [upload=471506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:06:29.798186648 (537h37m29.669269683s) [seg_pos={0 0}, upload=471506]: Begin upload segment... Dec 28 01:06:29.798237866 (537h37m29.669320941s) [seg_pos={0 0}, upload=471506]: Done begin upload segment. Dec 28 01:06:29.798292840 (537h37m29.669375875s) [noise=true , address=145.79.227.11:4731 , piece_id=YHJ5KGPWPQVLD4RVJXFMMNODXHCYX2CCPCGQDF2WASQSVHDFSVEQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.798304803 (537h37m29.669387838s) [noise=true , address=77.90.31.10:4517 , piece_id=AZCYGK7ZSALKHNV3THLF56AM5RMORCTJUCA2YOACYOEEBPQZUQ4A, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.798805466 (537h37m29.669888511s) [noise=true , address=145.79.227.46:4603 , piece_id=KV3WWPOSNPOXREXWKK22P2S3MURAOIU6UDSAJZU5YD5VDDQ7LS2Q, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.799191601 (537h37m29.670274636s) [noise=true , address=145.79.227.39:4014 , piece_id=PDH3T2IIT47HD2ELF3IOB4UGBPVPM66KE2EUJIHKAMGU2WS3DIZA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.799604598 (537h37m29.670687633s) [noise=true , address=77.90.31.13:4627 , piece_id=COLW6V55W7BAH3QKDPFKGRNMKEPUNXJZEOO337V246Q2CH7JAP4A, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800069864 (537h37m29.6711529s) [noise=true , address=145.79.227.45:4231 , piece_id=MC4QFPM7CX56ASQBYWMARUJCM77DE77A26LKXZXJDCKQZHDS7BAA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800540660 (537h37m29.671623695s) [noise=true , address=45.91.116.4:4939 , piece_id=CVRDL5CUWHVEH6XH4H3DIFZBFSE6D3IDSIYYPIRXD5P2B6XK4IEQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800948367 (537h37m29.672031412s) [noise=true , address=145.79.227.52:4446 , piece_id=3UXUTOXVHKWNKZJNMO3N3DFP2R6FAIFP24O4I4AXI6WWYO7MIUWA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801347978 (537h37m29.672431013s) [noise=true , address=45.91.116.13:4156 , piece_id=RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801734002 (537h37m29.672817048s) [noise=true , address=145.79.227.47:4266 , piece_id=ZMCJUICWQKQPHHNSNJLJB3D6YSOUCM6VZTNNTJTFDXAVHYS3S6TQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801898556 (537h37m29.672981601s) [noise=true , address=77.90.31.11:4556 , piece_id=VBR5YYAC7SF7RKO67CTRMOQ5ZT3TPE7ZUFXNDUXXBSZIPVIXHLQA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.802371667 (537h37m29.673454712s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.802698579 (537h37m29.673781625s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.803122547 (537h37m29.674205592s) [noise=true , address=77.90.31.14:4663 , piece_id=NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.803604314 (537h37m29.674687349s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.809721563 (537h37m29.680804598s) [noise=true , address=45.91.116.6:4975 , piece_id=PUMZ6TTLVNWBKD66TW7KSIFMOOGMGQ7ZJ54BQVGTPNFZDFLMGVQA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:31.533662902 (537h37m31.404745957s) [upload=471506]: Next returned nil segment Dec 28 01:06:31.533679524 (537h37m31.404762559s) [upload=471506]: Waiting for error group managing segments... Dec 28 01:06:31.828015661 (537h37m31.699098706s) [noise=true , address=145.79.227.46:4603 , piece_id=KV3WWPOSNPOXREXWKK22P2S3MURAOIU6UDSAJZU5YD5VDDQ7LS2Q, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.833605936 (537h37m31.704688981s) [noise=true , address=145.79.227.39:4014 , piece_id=PDH3T2IIT47HD2ELF3IOB4UGBPVPM66KE2EUJIHKAMGU2WS3DIZA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.836764400 (537h37m31.707847445s) [noise=true , address=145.79.227.11:4731 , piece_id=YHJ5KGPWPQVLD4RVJXFMMNODXHCYX2CCPCGQDF2WASQSVHDFSVEQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.840430000 (537h37m31.711513035s) [noise=true , address=145.79.227.45:4231 , piece_id=MC4QFPM7CX56ASQBYWMARUJCM77DE77A26LKXZXJDCKQZHDS7BAA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.840442403 (537h37m31.711525448s) [noise=true , address=145.79.227.47:4266 , piece_id=ZMCJUICWQKQPHHNSNJLJB3D6YSOUCM6VZTNNTJTFDXAVHYS3S6TQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.869641091 (537h37m31.740724126s) [noise=true , address=145.79.227.52:4446 , piece_id=3UXUTOXVHKWNKZJNMO3N3DFP2R6FAIFP24O4I4AXI6WWYO7MIUWA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.025232312 (537h37m31.896315358s) [noise=true , address=77.90.31.11:4556 , piece_id=VBR5YYAC7SF7RKO67CTRMOQ5ZT3TPE7ZUFXNDUXXBSZIPVIXHLQA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.025868604 (537h37m31.896951649s) [noise=true , address=77.90.31.13:4627 , piece_id=COLW6V55W7BAH3QKDPFKGRNMKEPUNXJZEOO337V246Q2CH7JAP4A, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.059586988 (537h37m31.930670033s) [noise=true , address=77.90.31.14:4663 , piece_id=NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.074009770 (537h37m31.945092815s) [noise=true , address=45.91.116.6:4975 , piece_id=PUMZ6TTLVNWBKD66TW7KSIFMOOGMGQ7ZJ54BQVGTPNFZDFLMGVQA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.076623106 (537h37m31.947706141s) [noise=true , address=77.90.31.10:4517 , piece_id=AZCYGK7ZSALKHNV3THLF56AM5RMORCTJUCA2YOACYOEEBPQZUQ4A, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.111222067 (537h37m31.982305102s) [noise=true , address=45.91.116.4:4939 , piece_id=CVRDL5CUWHVEH6XH4H3DIFZBFSE6D3IDSIYYPIRXD5P2B6XK4IEQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.193481548 (537h37m32.064564583s) [noise=true , address=45.91.116.13:4156 , piece_id=RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.193513679 (537h37m32.064596724s) [seg_pos={0 0}, upload=471506]: Segment reached optimal threshold of 13 pieces. Dec 28 01:06:32.193735521 (537h37m32.064818566s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:32.193742925 (537h37m32.06482597s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194041604 (537h37m32.065124649s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection Dec 28 01:06:32.194049309 (537h37m32.065132344s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194222388 (537h37m32.065305443s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:32.194244140 (537h37m32.065327195s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194278274 (537h37m32.065361319s) [seg_pos={0 0}, upload=471506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:06:32.194499506 (537h37m32.065582551s) [upload=471506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:06:32.194505918 (537h37m32.065588963s) [upload=471506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:06:32.496398003 (537h37m32.367481038s) [upload=471506]: Done uploading segments... Dec 28 01:06:32.496415877 (537h37m32.367498922s) [upload=471506]: Upload finished. err: <nil> Duration: 18.115310801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXXX] duration: 30.002846976s trace: https://jaeger.dev.storj.io/trace/30a26f0522322093 FAIL (2025-12-28 01:06:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA) context deadline exceeded ======LOGS====== :06:29.553789671 (537h37m29.424872716s) [upload=471506]: Starting upload Dec 28 01:06:29.553880263 (537h37m29.424963298s) [upload=471506]: Done starting upload Dec 28 01:06:29.553926091 (537h37m29.425009126s) [upload=471506]: Uploading segments... Dec 28 01:06:29.554081627 (537h37m29.425164662s) [upload=471506]: Got next segment. Inline: false Dec 28 01:06:29.554126372 (537h37m29.425209407s) [upload=471506]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:06:29.554132474 (537h37m29.425215519s) [upload=471506]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:06:29.798186648 (537h37m29.669269683s) [seg_pos={0 0}, upload=471506]: Begin upload segment... Dec 28 01:06:29.798237866 (537h37m29.669320941s) [seg_pos={0 0}, upload=471506]: Done begin upload segment. Dec 28 01:06:29.798292840 (537h37m29.669375875s) [noise=true , address=145.79.227.11:4731 , piece_id=YHJ5KGPWPQVLD4RVJXFMMNODXHCYX2CCPCGQDF2WASQSVHDFSVEQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.798304803 (537h37m29.669387838s) [noise=true , address=77.90.31.10:4517 , piece_id=AZCYGK7ZSALKHNV3THLF56AM5RMORCTJUCA2YOACYOEEBPQZUQ4A, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.798805466 (537h37m29.669888511s) [noise=true , address=145.79.227.46:4603 , piece_id=KV3WWPOSNPOXREXWKK22P2S3MURAOIU6UDSAJZU5YD5VDDQ7LS2Q, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.799191601 (537h37m29.670274636s) [noise=true , address=145.79.227.39:4014 , piece_id=PDH3T2IIT47HD2ELF3IOB4UGBPVPM66KE2EUJIHKAMGU2WS3DIZA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.799604598 (537h37m29.670687633s) [noise=true , address=77.90.31.13:4627 , piece_id=COLW6V55W7BAH3QKDPFKGRNMKEPUNXJZEOO337V246Q2CH7JAP4A, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800069864 (537h37m29.6711529s) [noise=true , address=145.79.227.45:4231 , piece_id=MC4QFPM7CX56ASQBYWMARUJCM77DE77A26LKXZXJDCKQZHDS7BAA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800540660 (537h37m29.671623695s) [noise=true , address=45.91.116.4:4939 , piece_id=CVRDL5CUWHVEH6XH4H3DIFZBFSE6D3IDSIYYPIRXD5P2B6XK4IEQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.800948367 (537h37m29.672031412s) [noise=true , address=145.79.227.52:4446 , piece_id=3UXUTOXVHKWNKZJNMO3N3DFP2R6FAIFP24O4I4AXI6WWYO7MIUWA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801347978 (537h37m29.672431013s) [noise=true , address=45.91.116.13:4156 , piece_id=RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801734002 (537h37m29.672817048s) [noise=true , address=145.79.227.47:4266 , piece_id=ZMCJUICWQKQPHHNSNJLJB3D6YSOUCM6VZTNNTJTFDXAVHYS3S6TQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.801898556 (537h37m29.672981601s) [noise=true , address=77.90.31.11:4556 , piece_id=VBR5YYAC7SF7RKO67CTRMOQ5ZT3TPE7ZUFXNDUXXBSZIPVIXHLQA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.802371667 (537h37m29.673454712s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.802698579 (537h37m29.673781625s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.803122547 (537h37m29.674205592s) [noise=true , address=77.90.31.14:4663 , piece_id=NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.803604314 (537h37m29.674687349s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:29.809721563 (537h37m29.680804598s) [noise=true , address=45.91.116.6:4975 , piece_id=PUMZ6TTLVNWBKD66TW7KSIFMOOGMGQ7ZJ54BQVGTPNFZDFLMGVQA, seg_pos={0 0}, upload=471506]: Uploading piece... Dec 28 01:06:31.533662902 (537h37m31.404745957s) [upload=471506]: Next returned nil segment Dec 28 01:06:31.533679524 (537h37m31.404762559s) [upload=471506]: Waiting for error group managing segments... Dec 28 01:06:31.828015661 (537h37m31.699098706s) [noise=true , address=145.79.227.46:4603 , piece_id=KV3WWPOSNPOXREXWKK22P2S3MURAOIU6UDSAJZU5YD5VDDQ7LS2Q, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.833605936 (537h37m31.704688981s) [noise=true , address=145.79.227.39:4014 , piece_id=PDH3T2IIT47HD2ELF3IOB4UGBPVPM66KE2EUJIHKAMGU2WS3DIZA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.836764400 (537h37m31.707847445s) [noise=true , address=145.79.227.11:4731 , piece_id=YHJ5KGPWPQVLD4RVJXFMMNODXHCYX2CCPCGQDF2WASQSVHDFSVEQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.840430000 (537h37m31.711513035s) [noise=true , address=145.79.227.45:4231 , piece_id=MC4QFPM7CX56ASQBYWMARUJCM77DE77A26LKXZXJDCKQZHDS7BAA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.840442403 (537h37m31.711525448s) [noise=true , address=145.79.227.47:4266 , piece_id=ZMCJUICWQKQPHHNSNJLJB3D6YSOUCM6VZTNNTJTFDXAVHYS3S6TQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:31.869641091 (537h37m31.740724126s) [noise=true , address=145.79.227.52:4446 , piece_id=3UXUTOXVHKWNKZJNMO3N3DFP2R6FAIFP24O4I4AXI6WWYO7MIUWA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.025232312 (537h37m31.896315358s) [noise=true , address=77.90.31.11:4556 , piece_id=VBR5YYAC7SF7RKO67CTRMOQ5ZT3TPE7ZUFXNDUXXBSZIPVIXHLQA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.025868604 (537h37m31.896951649s) [noise=true , address=77.90.31.13:4627 , piece_id=COLW6V55W7BAH3QKDPFKGRNMKEPUNXJZEOO337V246Q2CH7JAP4A, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.059586988 (537h37m31.930670033s) [noise=true , address=77.90.31.14:4663 , piece_id=NV4EUGFAKEG46MQWCMXMNV5NA3AI34APWEDNDOHSEZY5ZUEKWZYA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.074009770 (537h37m31.945092815s) [noise=true , address=45.91.116.6:4975 , piece_id=PUMZ6TTLVNWBKD66TW7KSIFMOOGMGQ7ZJ54BQVGTPNFZDFLMGVQA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.076623106 (537h37m31.947706141s) [noise=true , address=77.90.31.10:4517 , piece_id=AZCYGK7ZSALKHNV3THLF56AM5RMORCTJUCA2YOACYOEEBPQZUQ4A, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.111222067 (537h37m31.982305102s) [noise=true , address=45.91.116.4:4939 , piece_id=CVRDL5CUWHVEH6XH4H3DIFZBFSE6D3IDSIYYPIRXD5P2B6XK4IEQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.193481548 (537h37m32.064564583s) [noise=true , address=45.91.116.13:4156 , piece_id=RK3DSKFCY3T45MEUEWJGMKAM4L6YGMRBYRDHHNTD3HCJQIQB7KKA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: <nil> Dec 28 01:06:32.193513679 (537h37m32.064596724s) [seg_pos={0 0}, upload=471506]: Segment reached optimal threshold of 13 pieces. Dec 28 01:06:32.193735521 (537h37m32.064818566s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:32.193742925 (537h37m32.06482597s) [noise=true , address=45.91.116.9:4263 , piece_id=6WDYAERY6TMOAK7AZMHTBIKC3ZV356EZ4KHV7GRLPTOLFJAW7TFQ, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194041604 (537h37m32.065124649s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection; write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34942->145.79.227.41:4087: use of closed network connection Dec 28 01:06:32.194049309 (537h37m32.065132344s) [noise=true , address=145.79.227.41:4087 , piece_id=X4FZCXZEJDQJXUEZHDAE23FMPURMR23MFCNY2ZLOJAN47WFBX3PA, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194222388 (537h37m32.065305443s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:32.194244140 (537h37m32.065327195s) [noise=true , address=45.91.116.7:4012 , piece_id=73PHCZP3AB5MUXOUOBENI7UL45D2E6BJ5YNG2SGZVNS4S4N4YXMQ, seg_pos={0 0}, upload=471506]: Long tail cancellation due to optimal threshold Dec 28 01:06:32.194278274 (537h37m32.065361319s) [seg_pos={0 0}, upload=471506]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:06:32.194499506 (537h37m32.065582551s) [upload=471506]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:06:32.194505918 (537h37m32.065588963s) [upload=471506]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:06:32.496398003 (537h37m32.367481038s) [upload=471506]: Done uploading segments... Dec 28 01:06:32.496415877 (537h37m32.367498922s) [upload=471506]: Upload finished. err: <nil> Duration: 30.002857646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XXX] duration: 30.003656521s trace: https://jaeger.dev.storj.io/trace/28dc0195de2da742 FAIL (2025-12-28 01:06:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: XGNKZ4RRXTNFGXY4XH63QSKM6DS6IDBI3CYKNDSN4WX65GRSBFRA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: PUTNIHJ5REFFNHFVISNPPLQ5FDPG7JHPWGKP4VDBVTPL7NCP726A) context deadline exceeded ======LOGS====== Dec 28 01:05:59.331521158 (537h36m59.202604203s) [upload=471495]: Starting upload Dec 28 01:05:59.331597524 (537h36m59.202680569s) [upload=471495]: Done starting upload Dec 28 01:05:59.331672426 (537h36m59.202755461s) [upload=471495]: Uploading segments... Dec 28 01:05:59.331857278 (537h36m59.202940313s) [upload=471495]: Got next segment. Inline: false Dec 28 01:05:59.331893026 (537h36m59.202976081s) [upload=471495]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:05:59.331898085 (537h36m59.202981131s) [upload=471495]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:05:59.576405939 (537h36m59.447488974s) [seg_pos={0 0}, upload=471495]: Begin upload segment... Dec 28 01:05:59.576448109 (537h36m59.447531154s) [seg_pos={0 0}, upload=471495]: Done begin upload segment. Dec 28 01:05:59.576479779 (537h36m59.447562824s) [noise=true , address=145.79.227.48:4302 , piece_id=2M2IHTNCXKCY36XDHHMOGJWJM72GM4LJ3SCQVDRXLWVN2SAJHFXQ, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.576485681 (537h36m59.447568716s) [noise=true , address=145.79.227.19:4079 , piece_id=IOGFFG7BNBO75H2IW7AVU4BNQUVNQUYY7NLISEYR4ASUJNKIL2FA, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.577235919 (537h36m59.448318964s) [noise=true , address=145.79.227.5:4221 , piece_id=GIYTZH52C3F2MPJRTVT6W4LQKDBFJ6J22LMZBNHL73V3FKPA7JNA, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.577708660 (537h36m59.448791705s) [noise=true , address=145.79.227.30:4761 , piece_id=4ZLPRZUTFK63E3IUUER5T2RAPVOB765I33QQSCR23FCDLOR3FA4A, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.578108300 (537h36m59.449191335s) [noise=true , address=45.91.116.13:4154 , piece_id=RXCQDYXOTT6A44M5LED6WLAEJUPWN2R22WE3ZFNCFNTDYFUWVENQ, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.578121005 (537h36m59.44920404s) [noise=true , address=45.91.116.5:4299 , piece_id=PLRMBVNJ2HGORSIPBBK2T4S55JOKQTZRGMAIZAE6R5UIZWBE4SDA, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.578601570 (537h36m59.449684615s) [noise=true , address=145.79.227.52:4445 , piece_id=ZVJGXQMMYKKP42WNRR5JB7UDY7K6LRA56REDNNTNFCOTQYVCU4PA, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.578686772 (537h36m59.449769817s) [noise=true , address=45.91.116.9:4261 , piece_id=XGNKZ4RRXTNFGXY4XH63QSKM6DS6IDBI3CYKNDSN4WX65GRSBFRA, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.579016089 (537h36m59.450099134s) [noise=true , address=45.91.116.12:4228 , piece_id=PUTNIHJ5REFFNHFVISNPPLQ5FDPG7JHPWGKP4VDBVTPL7NCP726A, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.579260805 (537h36m59.45034385s) [noise=true , address=77.90.31.15:4700 , piece_id=NXWXVWZ73CZKX4EHYE3XOPCXB3G2QSRCZG3BPZNXOWLXNAVGHXJQ, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.579361617 (537h36m59.450444662s) [noise=true , address=145.79.227.49:4339 , piece_id=PLOPINBKT5QQBJZ37OSBEXQIW2XMD7JECQ3LC5EVSM3MESW2LAKQ, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.579738715 (537h36m59.45082176s) [noise=true , address=145.79.227.29:4943 , piece_id=O6CVCDKYRQWYTAC2E5IM6XZDWUG445C2NK5KU33JDCKWBSAWOSZA, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.579847202 (537h36m59.450930237s) [noise=true , address=145.79.227.13:4804 , piece_id=C3GKOKD7WAMUJFTV53NSI52BSMJJCALVVQ4VMPF6P7CAE46YXPHQ, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.580078132 (537h36m59.451161177s) [noise=true , address=145.79.227.16:4909 , piece_id=PMYS3UTE6TSJYMQFXQKOKBTH7OJYX6RUPPZ2ZWVKOI6TFINAM7RA, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.580350430 (537h36m59.451433475s) [noise=true , address=145.79.227.9:4365 , piece_id=VJ72CSCXWZYMBKLB45RE2RDWTF6QNA3UYDEYW6OYXNXT3ZJLT6MA, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:05:59.580435322 (537h36m59.451518357s) [noise=true , address=145.79.227.33:4870 , piece_id=EWZ7KDRZSJOI2DNE2MDM2NVUGX42FB6VHFB5WEHMP22AHKQ5AAHQ, seg_pos={0 0}, upload=471495]: Uploading piece... Dec 28 01:06:01.209123261 (537h37m1.080206306s) [upload=471495]: Next returned nil segment Dec 28 01:06:01.209147748 (537h37m1.080230803s) [upload=471495]: Waiting for error group managing segments... Dec 28 01:06:01.596879282 (537h37m1.467962327s) [noise=true , address=145.79.227.13:4804 , piece_id=C3GKOKD7WAMUJFTV53NSI52BSMJJCALVVQ4VMPF6P7CAE46YXPHQ, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.601279902 (537h37m1.472362957s) [noise=true , address=145.79.227.48:4302 , piece_id=2M2IHTNCXKCY36XDHHMOGJWJM72GM4LJ3SCQVDRXLWVN2SAJHFXQ, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.619636576 (537h37m1.490719621s) [noise=true , address=145.79.227.19:4079 , piece_id=IOGFFG7BNBO75H2IW7AVU4BNQUVNQUYY7NLISEYR4ASUJNKIL2FA, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.620476375 (537h37m1.49155942s) [noise=true , address=145.79.227.16:4909 , piece_id=PMYS3UTE6TSJYMQFXQKOKBTH7OJYX6RUPPZ2ZWVKOI6TFINAM7RA, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.621063183 (537h37m1.492146228s) [noise=true , address=145.79.227.9:4365 , piece_id=VJ72CSCXWZYMBKLB45RE2RDWTF6QNA3UYDEYW6OYXNXT3ZJLT6MA, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.623231742 (537h37m1.494314787s) [noise=true , address=145.79.227.5:4221 , piece_id=GIYTZH52C3F2MPJRTVT6W4LQKDBFJ6J22LMZBNHL73V3FKPA7JNA, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.639941039 (537h37m1.511024084s) [noise=true , address=145.79.227.52:4445 , piece_id=ZVJGXQMMYKKP42WNRR5JB7UDY7K6LRA56REDNNTNFCOTQYVCU4PA, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.764338533 (537h37m1.635421578s) [noise=true , address=77.90.31.15:4700 , piece_id=NXWXVWZ73CZKX4EHYE3XOPCXB3G2QSRCZG3BPZNXOWLXNAVGHXJQ, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.817762270 (537h37m1.688845315s) [noise=true , address=45.91.116.13:4154 , piece_id=RXCQDYXOTT6A44M5LED6WLAEJUPWN2R22WE3ZFNCFNTDYFUWVENQ, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.848807104 (537h37m1.719890149s) [noise=true , address=45.91.116.12:4228 , piece_id=PUTNIHJ5REFFNHFVISNPPLQ5FDPG7JHPWGKP4VDBVTPL7NCP726A, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.872114405 (537h37m1.74319746s) [noise=true , address=45.91.116.9:4261 , piece_id=XGNKZ4RRXTNFGXY4XH63QSKM6DS6IDBI3CYKNDSN4WX65GRSBFRA, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:01.872297444 (537h37m1.743380489s) [noise=true , address=45.91.116.5:4299 , piece_id=PLRMBVNJ2HGORSIPBBK2T4S55JOKQTZRGMAIZAE6R5UIZWBE4SDA, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:02.855954077 (537h37m2.727037112s) [noise=true , address=145.79.227.30:4761 , piece_id=4ZLPRZUTFK63E3IUUER5T2RAPVOB765I33QQSCR23FCDLOR3FA4A, seg_pos={0 0}, upload=471495]: Done uploading piece. err: <nil> Dec 28 01:06:02.855979115 (537h37m2.72706216s) [seg_pos={0 0}, upload=471495]: Segment reached optimal threshold of 13 pieces. Dec 28 01:06:02.856224322 (537h37m2.727307357s) [noise=true , address=145.79.227.49:4339 , piece_id=PLOPINBKT5QQBJZ37OSBEXQIW2XMD7JECQ3LC5EVSM3MESW2LAKQ, seg_pos={0 0}, upload=471495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:02.856231685 (537h37m2.72731473s) [noise=true , address=145.79.227.49:4339 , piece_id=PLOPINBKT5QQBJZ37OSBEXQIW2XMD7JECQ3LC5EVSM3MESW2LAKQ, seg_pos={0 0}, upload=471495]: Long tail cancellation due to optimal threshold Dec 28 01:06:02.856458257 (537h37m2.727541302s) [noise=true , address=145.79.227.29:4943 , piece_id=O6CVCDKYRQWYTAC2E5IM6XZDWUG445C2NK5KU33JDCKWBSAWOSZA, seg_pos={0 0}, upload=471495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:02.856464369 (537h37m2.727547404s) [noise=true , address=145.79.227.29:4943 , piece_id=O6CVCDKYRQWYTAC2E5IM6XZDWUG445C2NK5KU33JDCKWBSAWOSZA, seg_pos={0 0}, upload=471495]: Long tail cancellation due to optimal threshold Dec 28 01:06:02.856572074 (537h37m2.727655119s) [noise=true , address=145.79.227.33:4870 , piece_id=EWZ7KDRZSJOI2DNE2MDM2NVUGX42FB6VHFB5WEHMP22AHKQ5AAHQ, seg_pos={0 0}, upload=471495]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 01:06:02.856578396 (537h37m2.727661431s) [noise=true , address=145.79.227.33:4870 , piece_id=EWZ7KDRZSJOI2DNE2MDM2NVUGX42FB6VHFB5WEHMP22AHKQ5AAHQ, seg_pos={0 0}, upload=471495]: Long tail cancellation due to optimal threshold Dec 28 01:06:02.856595208 (537h37m2.727678243s) [seg_pos={0 0}, upload=471495]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:06:02.856675972 (537h37m2.727759007s) [upload=471495]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:06:02.856680901 (537h37m2.727763946s) [upload=471495]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:06:03.155562752 (537h37m3.026645797s) [upload=471495]: Done uploading segments... Dec 28 01:06:03.155582510 (537h37m3.026665545s) [upload=471495]: Upload finished. err: <nil> Duration: 30.003668904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 15.116310433s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 01:06:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 15.116332996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..XX] duration: 30.002880726s trace: https://jaeger.dev.storj.io/trace/2f825795ce5974fe FAIL (2025-12-28 01:05:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: NN5FAU7M2NWL3IVG2AW47IS742P4MBAOTKHF2GBXIHSS3PLA7RBA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: XDWPFKJ5TTBJBSEJDL7OJ72YKXDNCVYHL3MRNX3AC3QGHQV64BNA) context deadline exceeded ======LOGS====== 28 01:05:29.131989587 (537h36m29.003072653s) [upload=471488]: Starting upload Dec 28 01:05:29.132072755 (537h36m29.00315579s) [upload=471488]: Done starting upload Dec 28 01:05:29.132127199 (537h36m29.003210245s) [upload=471488]: Uploading segments... Dec 28 01:05:29.132427973 (537h36m29.003511018s) [upload=471488]: Got next segment. Inline: false Dec 28 01:05:29.132472237 (537h36m29.003555272s) [upload=471488]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:05:29.132479019 (537h36m29.003562054s) [upload=471488]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:05:29.378486799 (537h36m29.249569834s) [seg_pos={0 0}, upload=471488]: Begin upload segment... Dec 28 01:05:29.378538317 (537h36m29.249621362s) [seg_pos={0 0}, upload=471488]: Done begin upload segment. Dec 28 01:05:29.378565638 (537h36m29.249648685s) [noise=true , address=145.79.227.15:4876 , piece_id=HOB36YO6QBJHMCYWIXFW4263NVJXRAKUQV2K4LWTZF5YAVELKMCQ, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.378573654 (537h36m29.24965669s) [noise=true , address=145.79.227.42:4121 , piece_id=Y6CVTFEKGH5RRON7PUMBNDJQNXCMNHOYT6IR4SREM4DL5FEDASSQ, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.379187733 (537h36m29.250270778s) [noise=true , address=45.91.116.6:4975 , piece_id=NY6LBJBYA5MRZSAQUWO6DONTV5JDMJO6Y4SP6LIURL6BLSL52O5A, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.379582696 (537h36m29.250665731s) [noise=true , address=145.79.227.2:4114 , piece_id=6FXWAJX7NFOR7RMB76IAN76QUHCEPHNIA2DILQCIUJJT6DWT2P6Q, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.379890763 (537h36m29.250973798s) [noise=true , address=145.79.227.47:4266 , piece_id=IDX6TPUSJJEXRJ32RKEDWODFSDPBPD2442OPMWLMJMLMF57T2R4Q, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.380211553 (537h36m29.251294598s) [noise=true , address=77.90.31.12:4589 , piece_id=NN5FAU7M2NWL3IVG2AW47IS742P4MBAOTKHF2GBXIHSS3PLA7RBA, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.380222744 (537h36m29.251305779s) [noise=true , address=45.91.116.11:4118 , piece_id=YBL3U7CDRBLKYVVOP42R6HQNXM2BJXHO6U7LFYYYD2YVQVAYKFZA, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.380522626 (537h36m29.251605661s) [noise=true , address=145.79.227.11:4731 , piece_id=XS6UISUL5MGERUAK5RDI2Q6U5DBU33H24YOEWLFCDMTG3JPKKNUA, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.380811626 (537h36m29.251894661s) [noise=true , address=45.91.116.5:4300 , piece_id=4CSLFSTNBWDG4O6VD6SU3ERSWNMZ2CEGO7LJOX6SAO2WGZQ5BKJQ, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.381141955 (537h36m29.25222499s) [noise=true , address=45.91.116.10:4082 , piece_id=NL2GAJLJTUJIOCILVFFKSNOAERWALEIVUQSP5ASBMBZH3J6DPQ3A, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.381774049 (537h36m29.252857094s) [noise=true , address=77.90.31.15:4700 , piece_id=NOZ3ZCJKOHXW5ZBJXLYJL7AVXY37PI2TRF26A3UVMGPSC3KD3OIQ, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.382933877 (537h36m29.254016922s) [noise=true , address=45.91.116.9:4264 , piece_id=XDWPFKJ5TTBJBSEJDL7OJ72YKXDNCVYHL3MRNX3AC3QGHQV64BNA, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.384404828 (537h36m29.255487963s) [noise=true , address=145.79.227.48:4302 , piece_id=VDUUWRT4XXQQWG4ZRXRDHKOZOCWN2W4ZL6V6WMSM42EQDVWQJM7A, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.388129891 (537h36m29.259212926s) [noise=true , address=145.79.227.19:4079 , piece_id=OBSW5MHJYHUTSEC23GGCLCNF2RA3BTVVDGL7U4VPGYUWIIWCZMRQ, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.391090829 (537h36m29.262173874s) [noise=true , address=77.90.31.13:4626 , piece_id=T374LK6UILYYKG3MLYZH4EUYTHROGHTTKWVC3ZZWFCLLAOKIRNVA, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:29.398077383 (537h36m29.269160418s) [noise=true , address=145.79.227.50:4374 , piece_id=BGB3VPX56GSIL5KMKGQTDEXXBFBBHAVJVLAX4C5UTVB3FYZMGQRQ, seg_pos={0 0}, upload=471488]: Uploading piece... Dec 28 01:05:31.004673977 (537h36m30.875757012s) [upload=471488]: Next returned nil segment Dec 28 01:05:31.004690759 (537h36m30.875773794s) [upload=471488]: Waiting for error group managing segments... Dec 28 01:05:31.389771635 (537h36m31.260854671s) [noise=true , address=145.79.227.11:4731 , piece_id=XS6UISUL5MGERUAK5RDI2Q6U5DBU33H24YOEWLFCDMTG3JPKKNUA, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.396482013 (537h36m31.267565048s) [noise=true , address=145.79.227.47:4266 , piece_id=IDX6TPUSJJEXRJ32RKEDWODFSDPBPD2442OPMWLMJMLMF57T2R4Q, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.412876992 (537h36m31.283960027s) [noise=true , address=145.79.227.15:4876 , piece_id=HOB36YO6QBJHMCYWIXFW4263NVJXRAKUQV2K4LWTZF5YAVELKMCQ, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.419470046 (537h36m31.290553081s) [noise=true , address=145.79.227.42:4121 , piece_id=Y6CVTFEKGH5RRON7PUMBNDJQNXCMNHOYT6IR4SREM4DL5FEDASSQ, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.420638131 (537h36m31.291721176s) [noise=true , address=145.79.227.19:4079 , piece_id=OBSW5MHJYHUTSEC23GGCLCNF2RA3BTVVDGL7U4VPGYUWIIWCZMRQ, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.421298348 (537h36m31.292381383s) [noise=true , address=145.79.227.50:4374 , piece_id=BGB3VPX56GSIL5KMKGQTDEXXBFBBHAVJVLAX4C5UTVB3FYZMGQRQ, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.430996845 (537h36m31.30207989s) [noise=true , address=145.79.227.48:4302 , piece_id=VDUUWRT4XXQQWG4ZRXRDHKOZOCWN2W4ZL6V6WMSM42EQDVWQJM7A, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.628826715 (537h36m31.49990977s) [noise=true , address=45.91.116.6:4975 , piece_id=NY6LBJBYA5MRZSAQUWO6DONTV5JDMJO6Y4SP6LIURL6BLSL52O5A, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.651211309 (537h36m31.522294354s) [noise=true , address=45.91.116.11:4118 , piece_id=YBL3U7CDRBLKYVVOP42R6HQNXM2BJXHO6U7LFYYYD2YVQVAYKFZA, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.657955872 (537h36m31.529038908s) [noise=true , address=45.91.116.9:4264 , piece_id=XDWPFKJ5TTBJBSEJDL7OJ72YKXDNCVYHL3MRNX3AC3QGHQV64BNA, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.661405762 (537h36m31.532488797s) [noise=true , address=77.90.31.12:4589 , piece_id=NN5FAU7M2NWL3IVG2AW47IS742P4MBAOTKHF2GBXIHSS3PLA7RBA, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.662604084 (537h36m31.533687129s) [noise=true , address=77.90.31.13:4626 , piece_id=T374LK6UILYYKG3MLYZH4EUYTHROGHTTKWVC3ZZWFCLLAOKIRNVA, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.677612702 (537h36m31.548695747s) [noise=true , address=77.90.31.15:4700 , piece_id=NOZ3ZCJKOHXW5ZBJXLYJL7AVXY37PI2TRF26A3UVMGPSC3KD3OIQ, seg_pos={0 0}, upload=471488]: Done uploading piece. err: <nil> Dec 28 01:05:31.677636988 (537h36m31.548720043s) [seg_pos={0 0}, upload=471488]: Segment reached optimal threshold of 13 pieces. Dec 28 01:05:31.677905831 (537h36m31.548988866s) [noise=true , address=45.91.116.5:4300 , piece_id=4CSLFSTNBWDG4O6VD6SU3ERSWNMZ2CEGO7LJOX6SAO2WGZQ5BKJQ, seg_pos={0 0}, upload=471488]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 01:05:31.677913725 (537h36m31.54899677s) [noise=true , address=45.91.116.5:4300 , piece_id=4CSLFSTNBWDG4O6VD6SU3ERSWNMZ2CEGO7LJOX6SAO2WGZQ5BKJQ, seg_pos={0 0}, upload=471488]: Long tail cancellation due to optimal threshold Dec 28 01:05:31.678034927 (537h36m31.549117962s) [noise=true , address=145.79.227.2:4114 , piece_id=6FXWAJX7NFOR7RMB76IAN76QUHCEPHNIA2DILQCIUJJT6DWT2P6Q, seg_pos={0 0}, upload=471488]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:40142->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:40142->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40142->145.79.227.2:4114: use of closed network connection Dec 28 01:05:31.678040247 (537h36m31.549123282s) [noise=true , address=145.79.227.2:4114 , piece_id=6FXWAJX7NFOR7RMB76IAN76QUHCEPHNIA2DILQCIUJJT6DWT2P6Q, seg_pos={0 0}, upload=471488]: Long tail cancellation due to optimal threshold Dec 28 01:05:31.678179782 (537h36m31.549262817s) [noise=true , address=45.91.116.10:4082 , piece_id=NL2GAJLJTUJIOCILVFFKSNOAERWALEIVUQSP5ASBMBZH3J6DPQ3A, seg_pos={0 0}, upload=471488]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 01:05:31.678187767 (537h36m31.549270802s) [noise=true , address=45.91.116.10:4082 , piece_id=NL2GAJLJTUJIOCILVFFKSNOAERWALEIVUQSP5ASBMBZH3J6DPQ3A, seg_pos={0 0}, upload=471488]: Long tail cancellation due to optimal threshold Dec 28 01:05:31.678204529 (537h36m31.549287574s) [seg_pos={0 0}, upload=471488]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:05:31.678266587 (537h36m31.549349622s) [upload=471488]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:05:31.678271536 (537h36m31.549354581s) [upload=471488]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:05:31.973772620 (537h36m31.844855715s) [upload=471488]: Done uploading segments... Dec 28 01:05:31.973836411 (537h36m31.844919456s) [upload=471488]: Upload finished. err: <nil> Duration: 30.002904369s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 22.446217838s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 01:05:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 22.446248205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..X] duration: 30.002725541s trace: https://jaeger.dev.storj.io/trace/50aacbceec15248d FAIL (2025-12-28 01:05:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: CNTF6H5BWIPGLU2NIIJTDM42NCE3WLSRT7VUGWY36WF3DV6DXRZQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: YHHQE4VZQ3FFTRGN2KFBBG47LYC3DWB3BBQ6EW6M5PZL66IWEDCQ) context deadline exceeded ======LOGS====== Dec 28 01:04:37.907890739 (537h35m37.778973774s) [upload=471475]: Starting upload Dec 28 01:04:37.907958217 (537h35m37.779041262s) [upload=471475]: Done starting upload Dec 28 01:04:37.908138160 (537h35m37.779221205s) [upload=471475]: Uploading segments... Dec 28 01:04:37.908331959 (537h35m37.779415004s) [upload=471475]: Got next segment. Inline: false Dec 28 01:04:37.908395259 (537h35m37.779478314s) [upload=471475]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:04:37.908402082 (537h35m37.779485117s) [upload=471475]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:04:38.157894807 (537h35m38.028977842s) [seg_pos={0 0}, upload=471475]: Begin upload segment... Dec 28 01:04:38.157937458 (537h35m38.029020493s) [seg_pos={0 0}, upload=471475]: Done begin upload segment. Dec 28 01:04:38.157962336 (537h35m38.029045371s) [noise=true , address=145.79.227.39:4013 , piece_id=3NUQXTEBLDQU5RNBTAJO5YANQEIRN5TZZKVIORK3JJMWRECZBXRA, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.157969820 (537h35m38.029052865s) [noise=true , address=145.79.227.33:4870 , piece_id=XSI2NWWOBT7IBGNEUWUUVNVFSK5IFBD7FOR3DKJ2PFUCUPKLTLHA, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.158641288 (537h35m38.029724323s) [noise=true , address=145.79.227.51:4410 , piece_id=YGG5UNQN2HSOQZ2PQN7BZVH2ACHDSGKNNBW2GZ3M2SQEWQ4M2WPA, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.158999049 (537h35m38.030082084s) [noise=true , address=145.79.227.16:4909 , piece_id=SOPLHG4AY6KPM7Y75NKYQLPJILHQE57VXRQBWTPGWAHQ6H25FDRQ, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.159007616 (537h35m38.030090661s) [noise=true , address=145.79.227.50:4375 , piece_id=3PGYN45MRI36OFLYLXPYO42Y42CVT457F7Y3ANYZKA4KCKVVK7FA, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.159363113 (537h35m38.030446158s) [noise=true , address=145.79.227.5:4223 , piece_id=MIDT23UE5ZWJPU2Q7CC4OLRWVCD25HQFXG7BTO6HQGMDNXW44GRQ, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.159596056 (537h35m38.030679091s) [noise=true , address=77.90.31.14:4661 , piece_id=PRPJBLLL53TJ3TF6CKFJKVT5ORUR3HNBOZOGMJYGV65JQXOEYIAQ, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.159642585 (537h35m38.03072563s) [noise=true , address=45.91.116.7:4011 , piece_id=3H7OIDARNPAIRPQ2VZ7MX3ADKUUPKV6ZEOS2PJ443ZA32A4FC47Q, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.159948087 (537h35m38.031031122s) [noise=true , address=45.91.116.11:4118 , piece_id=YHHQE4VZQ3FFTRGN2KFBBG47LYC3DWB3BBQ6EW6M5PZL66IWEDCQ, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.160276221 (537h35m38.031359256s) [noise=true , address=145.79.227.45:4231 , piece_id=FLRN6SEIAFP27P5M7RZW5BVEKZ7K7VSXDV2DFBQZC7PCD45FAWEA, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.160664641 (537h35m38.031747686s) [noise=true , address=45.91.116.10:4081 , piece_id=QITWQ4EUYBREY2FUHFHXHIJZILFTXWF3OA3F4PWSMBS72RJTAWDQ, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.161048923 (537h35m38.032131958s) [noise=true , address=145.79.227.41:4086 , piece_id=MMGU2PWM2HSJFM263K5K3C4QHY2WHLPDEIHIEEBOVIH6FHQQLLJA, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.161161166 (537h35m38.032244211s) [noise=true , address=145.79.227.47:4265 , piece_id=VMCEMKEJSBUCBCDXAO4M4SDMGUTAOJPUICT4URUPOWBQHN3K4L4Q, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.161716824 (537h35m38.032799869s) [noise=true , address=145.79.227.38:4979 , piece_id=TUZW72YZQLW4HSJVPLBMHWWEVGIUAV5EQX52ZR6YZN4YLLEJF5JQ, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.162135010 (537h35m38.033218056s) [noise=true , address=45.91.116.8:4048 , piece_id=CNTF6H5BWIPGLU2NIIJTDM42NCE3WLSRT7VUGWY36WF3DV6DXRZQ, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:38.162608082 (537h35m38.033691117s) [noise=true , address=45.91.116.4:4939 , piece_id=46MT3AYVEWOGIJEGC4F4NPDGIQ5SOTFLJ6DJRBX2D4QJIQGLYK5Q, seg_pos={0 0}, upload=471475]: Uploading piece... Dec 28 01:04:39.885209668 (537h35m39.756292703s) [upload=471475]: Next returned nil segment Dec 28 01:04:39.885265935 (537h35m39.75634898s) [upload=471475]: Waiting for error group managing segments... Dec 28 01:04:40.172760111 (537h35m40.043843156s) [noise=true , address=145.79.227.38:4979 , piece_id=TUZW72YZQLW4HSJVPLBMHWWEVGIUAV5EQX52ZR6YZN4YLLEJF5JQ, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.179842126 (537h35m40.050925161s) [noise=true , address=145.79.227.16:4909 , piece_id=SOPLHG4AY6KPM7Y75NKYQLPJILHQE57VXRQBWTPGWAHQ6H25FDRQ, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.182094225 (537h35m40.05317727s) [noise=true , address=145.79.227.50:4375 , piece_id=3PGYN45MRI36OFLYLXPYO42Y42CVT457F7Y3ANYZKA4KCKVVK7FA, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.183100240 (537h35m40.054183285s) [noise=true , address=145.79.227.45:4231 , piece_id=FLRN6SEIAFP27P5M7RZW5BVEKZ7K7VSXDV2DFBQZC7PCD45FAWEA, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.186773325 (537h35m40.05785636s) [noise=true , address=145.79.227.41:4086 , piece_id=MMGU2PWM2HSJFM263K5K3C4QHY2WHLPDEIHIEEBOVIH6FHQQLLJA, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.187096099 (537h35m40.058179134s) [noise=true , address=145.79.227.47:4265 , piece_id=VMCEMKEJSBUCBCDXAO4M4SDMGUTAOJPUICT4URUPOWBQHN3K4L4Q, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.194648370 (537h35m40.065731415s) [noise=true , address=145.79.227.39:4013 , piece_id=3NUQXTEBLDQU5RNBTAJO5YANQEIRN5TZZKVIORK3JJMWRECZBXRA, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.208532869 (537h35m40.079615914s) [noise=true , address=145.79.227.5:4223 , piece_id=MIDT23UE5ZWJPU2Q7CC4OLRWVCD25HQFXG7BTO6HQGMDNXW44GRQ, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.386139049 (537h35m40.257222094s) [noise=true , address=45.91.116.11:4118 , piece_id=YHHQE4VZQ3FFTRGN2KFBBG47LYC3DWB3BBQ6EW6M5PZL66IWEDCQ, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.416205771 (537h35m40.287288816s) [noise=true , address=45.91.116.7:4011 , piece_id=3H7OIDARNPAIRPQ2VZ7MX3ADKUUPKV6ZEOS2PJ443ZA32A4FC47Q, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.520156019 (537h35m40.391239064s) [noise=true , address=45.91.116.4:4939 , piece_id=46MT3AYVEWOGIJEGC4F4NPDGIQ5SOTFLJ6DJRBX2D4QJIQGLYK5Q, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:40.607642919 (537h35m40.478725954s) [noise=true , address=45.91.116.10:4081 , piece_id=QITWQ4EUYBREY2FUHFHXHIJZILFTXWF3OA3F4PWSMBS72RJTAWDQ, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:41.252891869 (537h35m41.123974904s) [noise=true , address=45.91.116.8:4048 , piece_id=CNTF6H5BWIPGLU2NIIJTDM42NCE3WLSRT7VUGWY36WF3DV6DXRZQ, seg_pos={0 0}, upload=471475]: Done uploading piece. err: <nil> Dec 28 01:04:41.252917698 (537h35m41.124000743s) [seg_pos={0 0}, upload=471475]: Segment reached optimal threshold of 13 pieces. Dec 28 01:04:41.253208501 (537h35m41.124291547s) [noise=true , address=77.90.31.14:4661 , piece_id=PRPJBLLL53TJ3TF6CKFJKVT5ORUR3HNBOZOGMJYGV65JQXOEYIAQ, seg_pos={0 0}, upload=471475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 28 01:04:41.253219203 (537h35m41.124302248s) [noise=true , address=77.90.31.14:4661 , piece_id=PRPJBLLL53TJ3TF6CKFJKVT5ORUR3HNBOZOGMJYGV65JQXOEYIAQ, seg_pos={0 0}, upload=471475]: Long tail cancellation due to optimal threshold Dec 28 01:04:41.253328761 (537h35m41.124411796s) [noise=true , address=145.79.227.51:4410 , piece_id=YGG5UNQN2HSOQZ2PQN7BZVH2ACHDSGKNNBW2GZ3M2SQEWQ4M2WPA, seg_pos={0 0}, upload=471475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; write tcp 188.166.225.170:49734->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:49734->145.79.227.51:4410: use of closed network connection; EOF Dec 28 01:04:41.253364990 (537h35m41.124448025s) [noise=true , address=145.79.227.51:4410 , piece_id=YGG5UNQN2HSOQZ2PQN7BZVH2ACHDSGKNNBW2GZ3M2SQEWQ4M2WPA, seg_pos={0 0}, upload=471475]: Long tail cancellation due to optimal threshold Dec 28 01:04:41.253591130 (537h35m41.124674165s) [noise=true , address=145.79.227.33:4870 , piece_id=XSI2NWWOBT7IBGNEUWUUVNVFSK5IFBD7FOR3DKJ2PFUCUPKLTLHA, seg_pos={0 0}, upload=471475]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 01:04:41.253598174 (537h35m41.124681219s) [noise=true , address=145.79.227.33:4870 , piece_id=XSI2NWWOBT7IBGNEUWUUVNVFSK5IFBD7FOR3DKJ2PFUCUPKLTLHA, seg_pos={0 0}, upload=471475]: Long tail cancellation due to optimal threshold Dec 28 01:04:41.253629414 (537h35m41.124712449s) [seg_pos={0 0}, upload=471475]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:04:41.253750644 (537h35m41.124833679s) [upload=471475]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:04:41.253757587 (537h35m41.124840632s) [upload=471475]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:04:41.548467724 (537h35m41.419550759s) [upload=471475]: Done uploading segments... Dec 28 01:04:41.548494235 (537h35m41.41957728s) [upload=471475]: Upload finished. err: <nil> Duration: 30.002748074s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 15.36044061s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 01:05:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 15.360773814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 21.480760539s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 01:04:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 21.480786829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X..] duration: 22.202218101s trace: https://jaeger.dev.storj.io/trace/23c9e3e621772eaa FAIL (2025-12-28 01:03:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 5RVRA653VU652RBIZD2LFHZSGM5Q5AK724VOVAGLCBMIVUAQLHPQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: MDCH2KR5YIEC6W5ZRQIUE24RDTFXQUQ7WC455URCSO2A24O2QQJA) context deadline exceeded ======LOGS====== Dec 28 01:02:10.826784102 (537h33m10.697867147s) [upload=471437]: Starting upload Dec 28 01:02:10.826894283 (537h33m10.697977318s) [upload=471437]: Done starting upload Dec 28 01:02:10.826963775 (537h33m10.69804681s) [upload=471437]: Uploading segments... Dec 28 01:02:10.827134290 (537h33m10.698217325s) [upload=471437]: Got next segment. Inline: false Dec 28 01:02:10.827186208 (537h33m10.698269243s) [upload=471437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:02:10.827191688 (537h33m10.698274724s) [upload=471437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:02:11.071003527 (537h33m10.942086562s) [seg_pos={0 0}, upload=471437]: Begin upload segment... Dec 28 01:02:11.071062219 (537h33m10.942145254s) [seg_pos={0 0}, upload=471437]: Done begin upload segment. Dec 28 01:02:11.071087568 (537h33m10.942170603s) [noise=true , address=45.91.116.11:4120 , piece_id=3B22HFCKRGGNGXDQIBZSBMGDJOOZ3LSMU6PTOUQFD5ZQ6M25O4JA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.071131622 (537h33m10.942214657s) [noise=true , address=45.91.116.13:4153 , piece_id=MDCH2KR5YIEC6W5ZRQIUE24RDTFXQUQ7WC455URCSO2A24O2QQJA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.071566620 (537h33m10.942649665s) [noise=true , address=45.91.116.5:4298 , piece_id=5RVRA653VU652RBIZD2LFHZSGM5Q5AK724VOVAGLCBMIVUAQLHPQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.071964077 (537h33m10.943047122s) [noise=true , address=45.91.116.14:4190 , piece_id=4QTQDGKAAELMJH7TEYPT5OQDFRNJEMM7HFUYH6YW7YDZWMXQ2SXA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.072381251 (537h33m10.943464286s) [noise=true , address=145.79.227.49:4338 , piece_id=5VCWN6CVAJR277DOAZAZ7CNUSVSDEA6WBSKDE5UGJQXK3E2FWBLA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.073570305 (537h33m10.94465334s) [noise=true , address=145.79.227.9:4367 , piece_id=ILWM6HCRLW6QPBJZOC66LNMYWPZEOFQSFTG5E73YBYUQAUDPVJWA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.074975762 (537h33m10.946058807s) [noise=true , address=145.79.227.14:4839 , piece_id=3Y3GBWR4ITF3ATAYKNP5RBSAOX3ZLRHB367ZWA33T3Q4AU4W754A, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.075100158 (537h33m10.946183203s) [noise=true , address=45.91.116.4:4937 , piece_id=WH2JE5KITZ7Z3O6QBL5XXOAPBZEMLN7Q7Q3H4MKEHUVBP2UEYUIQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.076709673 (537h33m10.947792718s) [noise=true , address=145.79.227.7:4295 , piece_id=IJ37FQ63XLMSY4IZB6WNJDLQILSDX7SUDWKRNLVWLZRDWUF7S4NQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.078710864 (537h33m10.949793909s) [noise=true , address=145.79.227.11:4729 , piece_id=6Y4BMMJQ434TJDUQ7RCY2L4DGEGO2VZGFBTBB6QWGCTY3KQQXRHQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.080373130 (537h33m10.951456175s) [noise=true , address=77.90.31.14:4662 , piece_id=SSZJ22VNDYOSYFIORHW7DF3M6HAPNXOK44NHSRLMU6P3XPFLRBBQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.081571140 (537h33m10.952654185s) [noise=true , address=145.79.227.5:4221 , piece_id=SNGQVGMSM46TBOZ4CKAXSBVZ3FY46UHYDDTBMTPR2O4PYZRGCLUA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.081790959 (537h33m10.952874004s) [noise=true , address=77.90.31.18:4805 , piece_id=TNON6SYKULOU2DKJCPBWS72KWC7S3BGNZCRDX2VBGCL5IQPUODFQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.083018066 (537h33m10.954101101s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYGTZ66P5T243DFWEJ5SC6JGKUCHYCF5UKWPMGYK4646EOX3MX3Q, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.083116252 (537h33m10.954199287s) [noise=true , address=77.90.31.19:4841 , piece_id=AVZTZGBSQTDACX63JHSIXSO6BAPYX7FIZQZI4XJXMJNKXMTFA4XQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.084099205 (537h33m10.95518225s) [noise=true , address=145.79.227.4:4187 , piece_id=F53SRWDDIQPGASETNA6ID4FJH65A65DHR5YB4IIV4S2BCT3T7UJA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:12.701033853 (537h33m12.572116898s) [upload=471437]: Next returned nil segment Dec 28 01:02:12.701052258 (537h33m12.572135303s) [upload=471437]: Waiting for error group managing segments... Dec 28 01:02:13.089970332 (537h33m12.961053367s) [noise=true , address=145.79.227.4:4187 , piece_id=F53SRWDDIQPGASETNA6ID4FJH65A65DHR5YB4IIV4S2BCT3T7UJA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.090592687 (537h33m12.961675722s) [noise=true , address=145.79.227.11:4729 , piece_id=6Y4BMMJQ434TJDUQ7RCY2L4DGEGO2VZGFBTBB6QWGCTY3KQQXRHQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.092987517 (537h33m12.964070552s) [noise=true , address=145.79.227.9:4367 , piece_id=ILWM6HCRLW6QPBJZOC66LNMYWPZEOFQSFTG5E73YBYUQAUDPVJWA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.093491116 (537h33m12.964574151s) [noise=true , address=145.79.227.7:4295 , piece_id=IJ37FQ63XLMSY4IZB6WNJDLQILSDX7SUDWKRNLVWLZRDWUF7S4NQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.094162414 (537h33m12.965245459s) [noise=true , address=145.79.227.5:4221 , piece_id=SNGQVGMSM46TBOZ4CKAXSBVZ3FY46UHYDDTBMTPR2O4PYZRGCLUA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.199130782 (537h33m13.070213827s) [noise=true , address=145.79.227.14:4839 , piece_id=3Y3GBWR4ITF3ATAYKNP5RBSAOX3ZLRHB367ZWA33T3Q4AU4W754A, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.255510518 (537h33m13.126593563s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYGTZ66P5T243DFWEJ5SC6JGKUCHYCF5UKWPMGYK4646EOX3MX3Q, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.283485757 (537h33m13.154568792s) [noise=true , address=45.91.116.5:4298 , piece_id=5RVRA653VU652RBIZD2LFHZSGM5Q5AK724VOVAGLCBMIVUAQLHPQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.326463233 (537h33m13.197546268s) [noise=true , address=45.91.116.14:4190 , piece_id=4QTQDGKAAELMJH7TEYPT5OQDFRNJEMM7HFUYH6YW7YDZWMXQ2SXA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.334510447 (537h33m13.205593482s) [noise=true , address=77.90.31.18:4805 , piece_id=TNON6SYKULOU2DKJCPBWS72KWC7S3BGNZCRDX2VBGCL5IQPUODFQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.358172643 (537h33m13.229255688s) [noise=true , address=45.91.116.4:4937 , piece_id=WH2JE5KITZ7Z3O6QBL5XXOAPBZEMLN7Q7Q3H4MKEHUVBP2UEYUIQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.417114919 (537h33m13.288197964s) [noise=true , address=45.91.116.13:4153 , piece_id=MDCH2KR5YIEC6W5ZRQIUE24RDTFXQUQ7WC455URCSO2A24O2QQJA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.421041465 (537h33m13.2921245s) [noise=true , address=45.91.116.11:4120 , piece_id=3B22HFCKRGGNGXDQIBZSBMGDJOOZ3LSMU6PTOUQFD5ZQ6M25O4JA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.421061955 (537h33m13.29214499s) [seg_pos={0 0}, upload=471437]: Segment reached optimal threshold of 13 pieces. Dec 28 01:02:13.421382966 (537h33m13.292466001s) [noise=true , address=77.90.31.19:4841 , piece_id=AVZTZGBSQTDACX63JHSIXSO6BAPYX7FIZQZI4XJXMJNKXMTFA4XQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 28 01:02:13.421396673 (537h33m13.292479708s) [noise=true , address=77.90.31.19:4841 , piece_id=AVZTZGBSQTDACX63JHSIXSO6BAPYX7FIZQZI4XJXMJNKXMTFA4XQ, seg_pos={0 0}, upload=471437]: Long tail cancellation due to optimal threshold Dec 28 01:02:13.421400319 (537h33m13.292483374s) [noise=true , address=77.90.31.14:4662 , piece_id=SSZJ22VNDYOSYFIORHW7DF3M6HAPNXOK44NHSRLMU6P3XPFLRBBQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): internal: context canceled Dec 28 01:02:13.421427261 (537h33m13.292510306s) [noise=true , address=77.90.31.14:4662 , piece_id=SSZJ22VNDYOSYFIORHW7DF3M6HAPNXOK44NHSRLMU6P3XPFLRBBQ, seg_pos={0 0}, upload=471437]: Long tail cancellation due to optimal threshold Dec 28 01:02:13.421768710 (537h33m13.292851765s) [noise=true , address=145.79.227.49:4338 , piece_id=5VCWN6CVAJR277DOAZAZ7CNUSVSDEA6WBSKDE5UGJQXK3E2FWBLA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 01:02:13.421778860 (537h33m13.292861905s) [noise=true , address=145.79.227.49:4338 , piece_id=5VCWN6CVAJR277DOAZAZ7CNUSVSDEA6WBSKDE5UGJQXK3E2FWBLA, seg_pos={0 0}, upload=471437]: Long tail cancellation due to optimal threshold Dec 28 01:02:13.421851118 (537h33m13.292934163s) [seg_pos={0 0}, upload=471437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:02:13.421969443 (537h33m13.293052488s) [upload=471437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:02:13.421977768 (537h33m13.293060813s) [upload=471437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:02:13.714293426 (537h33m13.585376461s) [upload=471437]: Done uploading segments... Dec 28 01:02:13.714312713 (537h33m13.585395748s) [upload=471437]: Upload finished. err: <nil> Duration: 22.202246586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X.] duration: 5.922640902s trace: https://jaeger.dev.storj.io/trace/23c9e3e621772eaa FAIL (2025-12-28 01:03:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 5RVRA653VU652RBIZD2LFHZSGM5Q5AK724VOVAGLCBMIVUAQLHPQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: MDCH2KR5YIEC6W5ZRQIUE24RDTFXQUQ7WC455URCSO2A24O2QQJA) context deadline exceeded ======LOGS====== Dec 28 01:02:10.826784102 (537h33m10.697867147s) [upload=471437]: Starting upload Dec 28 01:02:10.826894283 (537h33m10.697977318s) [upload=471437]: Done starting upload Dec 28 01:02:10.826963775 (537h33m10.69804681s) [upload=471437]: Uploading segments... Dec 28 01:02:10.827134290 (537h33m10.698217325s) [upload=471437]: Got next segment. Inline: false Dec 28 01:02:10.827186208 (537h33m10.698269243s) [upload=471437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:02:10.827191688 (537h33m10.698274724s) [upload=471437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:02:11.071003527 (537h33m10.942086562s) [seg_pos={0 0}, upload=471437]: Begin upload segment... Dec 28 01:02:11.071062219 (537h33m10.942145254s) [seg_pos={0 0}, upload=471437]: Done begin upload segment. Dec 28 01:02:11.071087568 (537h33m10.942170603s) [noise=true , address=45.91.116.11:4120 , piece_id=3B22HFCKRGGNGXDQIBZSBMGDJOOZ3LSMU6PTOUQFD5ZQ6M25O4JA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.071131622 (537h33m10.942214657s) [noise=true , address=45.91.116.13:4153 , piece_id=MDCH2KR5YIEC6W5ZRQIUE24RDTFXQUQ7WC455URCSO2A24O2QQJA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.071566620 (537h33m10.942649665s) [noise=true , address=45.91.116.5:4298 , piece_id=5RVRA653VU652RBIZD2LFHZSGM5Q5AK724VOVAGLCBMIVUAQLHPQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.071964077 (537h33m10.943047122s) [noise=true , address=45.91.116.14:4190 , piece_id=4QTQDGKAAELMJH7TEYPT5OQDFRNJEMM7HFUYH6YW7YDZWMXQ2SXA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.072381251 (537h33m10.943464286s) [noise=true , address=145.79.227.49:4338 , piece_id=5VCWN6CVAJR277DOAZAZ7CNUSVSDEA6WBSKDE5UGJQXK3E2FWBLA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.073570305 (537h33m10.94465334s) [noise=true , address=145.79.227.9:4367 , piece_id=ILWM6HCRLW6QPBJZOC66LNMYWPZEOFQSFTG5E73YBYUQAUDPVJWA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.074975762 (537h33m10.946058807s) [noise=true , address=145.79.227.14:4839 , piece_id=3Y3GBWR4ITF3ATAYKNP5RBSAOX3ZLRHB367ZWA33T3Q4AU4W754A, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.075100158 (537h33m10.946183203s) [noise=true , address=45.91.116.4:4937 , piece_id=WH2JE5KITZ7Z3O6QBL5XXOAPBZEMLN7Q7Q3H4MKEHUVBP2UEYUIQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.076709673 (537h33m10.947792718s) [noise=true , address=145.79.227.7:4295 , piece_id=IJ37FQ63XLMSY4IZB6WNJDLQILSDX7SUDWKRNLVWLZRDWUF7S4NQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.078710864 (537h33m10.949793909s) [noise=true , address=145.79.227.11:4729 , piece_id=6Y4BMMJQ434TJDUQ7RCY2L4DGEGO2VZGFBTBB6QWGCTY3KQQXRHQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.080373130 (537h33m10.951456175s) [noise=true , address=77.90.31.14:4662 , piece_id=SSZJ22VNDYOSYFIORHW7DF3M6HAPNXOK44NHSRLMU6P3XPFLRBBQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.081571140 (537h33m10.952654185s) [noise=true , address=145.79.227.5:4221 , piece_id=SNGQVGMSM46TBOZ4CKAXSBVZ3FY46UHYDDTBMTPR2O4PYZRGCLUA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.081790959 (537h33m10.952874004s) [noise=true , address=77.90.31.18:4805 , piece_id=TNON6SYKULOU2DKJCPBWS72KWC7S3BGNZCRDX2VBGCL5IQPUODFQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.083018066 (537h33m10.954101101s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYGTZ66P5T243DFWEJ5SC6JGKUCHYCF5UKWPMGYK4646EOX3MX3Q, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.083116252 (537h33m10.954199287s) [noise=true , address=77.90.31.19:4841 , piece_id=AVZTZGBSQTDACX63JHSIXSO6BAPYX7FIZQZI4XJXMJNKXMTFA4XQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.084099205 (537h33m10.95518225s) [noise=true , address=145.79.227.4:4187 , piece_id=F53SRWDDIQPGASETNA6ID4FJH65A65DHR5YB4IIV4S2BCT3T7UJA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:12.701033853 (537h33m12.572116898s) [upload=471437]: Next returned nil segment Dec 28 01:02:12.701052258 (537h33m12.572135303s) [upload=471437]: Waiting for error group managing segments... Dec 28 01:02:13.089970332 (537h33m12.961053367s) [noise=true , address=145.79.227.4:4187 , piece_id=F53SRWDDIQPGASETNA6ID4FJH65A65DHR5YB4IIV4S2BCT3T7UJA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.090592687 (537h33m12.961675722s) [noise=true , address=145.79.227.11:4729 , piece_id=6Y4BMMJQ434TJDUQ7RCY2L4DGEGO2VZGFBTBB6QWGCTY3KQQXRHQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.092987517 (537h33m12.964070552s) [noise=true , address=145.79.227.9:4367 , piece_id=ILWM6HCRLW6QPBJZOC66LNMYWPZEOFQSFTG5E73YBYUQAUDPVJWA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.093491116 (537h33m12.964574151s) [noise=true , address=145.79.227.7:4295 , piece_id=IJ37FQ63XLMSY4IZB6WNJDLQILSDX7SUDWKRNLVWLZRDWUF7S4NQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.094162414 (537h33m12.965245459s) [noise=true , address=145.79.227.5:4221 , piece_id=SNGQVGMSM46TBOZ4CKAXSBVZ3FY46UHYDDTBMTPR2O4PYZRGCLUA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.199130782 (537h33m13.070213827s) [noise=true , address=145.79.227.14:4839 , piece_id=3Y3GBWR4ITF3ATAYKNP5RBSAOX3ZLRHB367ZWA33T3Q4AU4W754A, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.255510518 (537h33m13.126593563s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYGTZ66P5T243DFWEJ5SC6JGKUCHYCF5UKWPMGYK4646EOX3MX3Q, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.283485757 (537h33m13.154568792s) [noise=true , address=45.91.116.5:4298 , piece_id=5RVRA653VU652RBIZD2LFHZSGM5Q5AK724VOVAGLCBMIVUAQLHPQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.326463233 (537h33m13.197546268s) [noise=true , address=45.91.116.14:4190 , piece_id=4QTQDGKAAELMJH7TEYPT5OQDFRNJEMM7HFUYH6YW7YDZWMXQ2SXA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.334510447 (537h33m13.205593482s) [noise=true , address=77.90.31.18:4805 , piece_id=TNON6SYKULOU2DKJCPBWS72KWC7S3BGNZCRDX2VBGCL5IQPUODFQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.358172643 (537h33m13.229255688s) [noise=true , address=45.91.116.4:4937 , piece_id=WH2JE5KITZ7Z3O6QBL5XXOAPBZEMLN7Q7Q3H4MKEHUVBP2UEYUIQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.417114919 (537h33m13.288197964s) [noise=true , address=45.91.116.13:4153 , piece_id=MDCH2KR5YIEC6W5ZRQIUE24RDTFXQUQ7WC455URCSO2A24O2QQJA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.421041465 (537h33m13.2921245s) [noise=true , address=45.91.116.11:4120 , piece_id=3B22HFCKRGGNGXDQIBZSBMGDJOOZ3LSMU6PTOUQFD5ZQ6M25O4JA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.421061955 (537h33m13.29214499s) [seg_pos={0 0}, upload=471437]: Segment reached optimal threshold of 13 pieces. Dec 28 01:02:13.421382966 (537h33m13.292466001s) [noise=true , address=77.90.31.19:4841 , piece_id=AVZTZGBSQTDACX63JHSIXSO6BAPYX7FIZQZI4XJXMJNKXMTFA4XQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 28 01:02:13.421396673 (537h33m13.292479708s) [noise=true , address=77.90.31.19:4841 , piece_id=AVZTZGBSQTDACX63JHSIXSO6BAPYX7FIZQZI4XJXMJNKXMTFA4XQ, seg_pos={0 0}, upload=471437]: Long tail cancellation due to optimal threshold Dec 28 01:02:13.421400319 (537h33m13.292483374s) [noise=true , address=77.90.31.14:4662 , piece_id=SSZJ22VNDYOSYFIORHW7DF3M6HAPNXOK44NHSRLMU6P3XPFLRBBQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): internal: context canceled Dec 28 01:02:13.421427261 (537h33m13.292510306s) [noise=true , address=77.90.31.14:4662 , piece_id=SSZJ22VNDYOSYFIORHW7DF3M6HAPNXOK44NHSRLMU6P3XPFLRBBQ, seg_pos={0 0}, upload=471437]: Long tail cancellation due to optimal threshold Dec 28 01:02:13.421768710 (537h33m13.292851765s) [noise=true , address=145.79.227.49:4338 , piece_id=5VCWN6CVAJR277DOAZAZ7CNUSVSDEA6WBSKDE5UGJQXK3E2FWBLA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 01:02:13.421778860 (537h33m13.292861905s) [noise=true , address=145.79.227.49:4338 , piece_id=5VCWN6CVAJR277DOAZAZ7CNUSVSDEA6WBSKDE5UGJQXK3E2FWBLA, seg_pos={0 0}, upload=471437]: Long tail cancellation due to optimal threshold Dec 28 01:02:13.421851118 (537h33m13.292934163s) [seg_pos={0 0}, upload=471437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:02:13.421969443 (537h33m13.293052488s) [upload=471437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:02:13.421977768 (537h33m13.293060813s) [upload=471437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:02:13.714293426 (537h33m13.585376461s) [upload=471437]: Done uploading segments... Dec 28 01:02:13.714312713 (537h33m13.585395748s) [upload=471437]: Upload finished. err: <nil> Duration: 5.922671882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.492739413s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 01:02:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 6.492768649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.X] duration: 30.002925027s trace: https://jaeger.dev.storj.io/trace/23c9e3e621772eaa FAIL (2025-12-28 01:02:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: 5RVRA653VU652RBIZD2LFHZSGM5Q5AK724VOVAGLCBMIVUAQLHPQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: MDCH2KR5YIEC6W5ZRQIUE24RDTFXQUQ7WC455URCSO2A24O2QQJA) context deadline exceeded ======LOGS====== Dec 28 01:02:10.826784102 (537h33m10.697867147s) [upload=471437]: Starting upload Dec 28 01:02:10.826894283 (537h33m10.697977318s) [upload=471437]: Done starting upload Dec 28 01:02:10.826963775 (537h33m10.69804681s) [upload=471437]: Uploading segments... Dec 28 01:02:10.827134290 (537h33m10.698217325s) [upload=471437]: Got next segment. Inline: false Dec 28 01:02:10.827186208 (537h33m10.698269243s) [upload=471437]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:02:10.827191688 (537h33m10.698274724s) [upload=471437]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:02:11.071003527 (537h33m10.942086562s) [seg_pos={0 0}, upload=471437]: Begin upload segment... Dec 28 01:02:11.071062219 (537h33m10.942145254s) [seg_pos={0 0}, upload=471437]: Done begin upload segment. Dec 28 01:02:11.071087568 (537h33m10.942170603s) [noise=true , address=45.91.116.11:4120 , piece_id=3B22HFCKRGGNGXDQIBZSBMGDJOOZ3LSMU6PTOUQFD5ZQ6M25O4JA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.071131622 (537h33m10.942214657s) [noise=true , address=45.91.116.13:4153 , piece_id=MDCH2KR5YIEC6W5ZRQIUE24RDTFXQUQ7WC455URCSO2A24O2QQJA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.071566620 (537h33m10.942649665s) [noise=true , address=45.91.116.5:4298 , piece_id=5RVRA653VU652RBIZD2LFHZSGM5Q5AK724VOVAGLCBMIVUAQLHPQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.071964077 (537h33m10.943047122s) [noise=true , address=45.91.116.14:4190 , piece_id=4QTQDGKAAELMJH7TEYPT5OQDFRNJEMM7HFUYH6YW7YDZWMXQ2SXA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.072381251 (537h33m10.943464286s) [noise=true , address=145.79.227.49:4338 , piece_id=5VCWN6CVAJR277DOAZAZ7CNUSVSDEA6WBSKDE5UGJQXK3E2FWBLA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.073570305 (537h33m10.94465334s) [noise=true , address=145.79.227.9:4367 , piece_id=ILWM6HCRLW6QPBJZOC66LNMYWPZEOFQSFTG5E73YBYUQAUDPVJWA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.074975762 (537h33m10.946058807s) [noise=true , address=145.79.227.14:4839 , piece_id=3Y3GBWR4ITF3ATAYKNP5RBSAOX3ZLRHB367ZWA33T3Q4AU4W754A, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.075100158 (537h33m10.946183203s) [noise=true , address=45.91.116.4:4937 , piece_id=WH2JE5KITZ7Z3O6QBL5XXOAPBZEMLN7Q7Q3H4MKEHUVBP2UEYUIQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.076709673 (537h33m10.947792718s) [noise=true , address=145.79.227.7:4295 , piece_id=IJ37FQ63XLMSY4IZB6WNJDLQILSDX7SUDWKRNLVWLZRDWUF7S4NQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.078710864 (537h33m10.949793909s) [noise=true , address=145.79.227.11:4729 , piece_id=6Y4BMMJQ434TJDUQ7RCY2L4DGEGO2VZGFBTBB6QWGCTY3KQQXRHQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.080373130 (537h33m10.951456175s) [noise=true , address=77.90.31.14:4662 , piece_id=SSZJ22VNDYOSYFIORHW7DF3M6HAPNXOK44NHSRLMU6P3XPFLRBBQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.081571140 (537h33m10.952654185s) [noise=true , address=145.79.227.5:4221 , piece_id=SNGQVGMSM46TBOZ4CKAXSBVZ3FY46UHYDDTBMTPR2O4PYZRGCLUA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.081790959 (537h33m10.952874004s) [noise=true , address=77.90.31.18:4805 , piece_id=TNON6SYKULOU2DKJCPBWS72KWC7S3BGNZCRDX2VBGCL5IQPUODFQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.083018066 (537h33m10.954101101s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYGTZ66P5T243DFWEJ5SC6JGKUCHYCF5UKWPMGYK4646EOX3MX3Q, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.083116252 (537h33m10.954199287s) [noise=true , address=77.90.31.19:4841 , piece_id=AVZTZGBSQTDACX63JHSIXSO6BAPYX7FIZQZI4XJXMJNKXMTFA4XQ, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:11.084099205 (537h33m10.95518225s) [noise=true , address=145.79.227.4:4187 , piece_id=F53SRWDDIQPGASETNA6ID4FJH65A65DHR5YB4IIV4S2BCT3T7UJA, seg_pos={0 0}, upload=471437]: Uploading piece... Dec 28 01:02:12.701033853 (537h33m12.572116898s) [upload=471437]: Next returned nil segment Dec 28 01:02:12.701052258 (537h33m12.572135303s) [upload=471437]: Waiting for error group managing segments... Dec 28 01:02:13.089970332 (537h33m12.961053367s) [noise=true , address=145.79.227.4:4187 , piece_id=F53SRWDDIQPGASETNA6ID4FJH65A65DHR5YB4IIV4S2BCT3T7UJA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.090592687 (537h33m12.961675722s) [noise=true , address=145.79.227.11:4729 , piece_id=6Y4BMMJQ434TJDUQ7RCY2L4DGEGO2VZGFBTBB6QWGCTY3KQQXRHQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.092987517 (537h33m12.964070552s) [noise=true , address=145.79.227.9:4367 , piece_id=ILWM6HCRLW6QPBJZOC66LNMYWPZEOFQSFTG5E73YBYUQAUDPVJWA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.093491116 (537h33m12.964574151s) [noise=true , address=145.79.227.7:4295 , piece_id=IJ37FQ63XLMSY4IZB6WNJDLQILSDX7SUDWKRNLVWLZRDWUF7S4NQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.094162414 (537h33m12.965245459s) [noise=true , address=145.79.227.5:4221 , piece_id=SNGQVGMSM46TBOZ4CKAXSBVZ3FY46UHYDDTBMTPR2O4PYZRGCLUA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.199130782 (537h33m13.070213827s) [noise=true , address=145.79.227.14:4839 , piece_id=3Y3GBWR4ITF3ATAYKNP5RBSAOX3ZLRHB367ZWA33T3Q4AU4W754A, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.255510518 (537h33m13.126593563s) [noise=true , address=145.79.227.42:4122 , piece_id=ZYGTZ66P5T243DFWEJ5SC6JGKUCHYCF5UKWPMGYK4646EOX3MX3Q, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.283485757 (537h33m13.154568792s) [noise=true , address=45.91.116.5:4298 , piece_id=5RVRA653VU652RBIZD2LFHZSGM5Q5AK724VOVAGLCBMIVUAQLHPQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.326463233 (537h33m13.197546268s) [noise=true , address=45.91.116.14:4190 , piece_id=4QTQDGKAAELMJH7TEYPT5OQDFRNJEMM7HFUYH6YW7YDZWMXQ2SXA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.334510447 (537h33m13.205593482s) [noise=true , address=77.90.31.18:4805 , piece_id=TNON6SYKULOU2DKJCPBWS72KWC7S3BGNZCRDX2VBGCL5IQPUODFQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.358172643 (537h33m13.229255688s) [noise=true , address=45.91.116.4:4937 , piece_id=WH2JE5KITZ7Z3O6QBL5XXOAPBZEMLN7Q7Q3H4MKEHUVBP2UEYUIQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.417114919 (537h33m13.288197964s) [noise=true , address=45.91.116.13:4153 , piece_id=MDCH2KR5YIEC6W5ZRQIUE24RDTFXQUQ7WC455URCSO2A24O2QQJA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.421041465 (537h33m13.2921245s) [noise=true , address=45.91.116.11:4120 , piece_id=3B22HFCKRGGNGXDQIBZSBMGDJOOZ3LSMU6PTOUQFD5ZQ6M25O4JA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: <nil> Dec 28 01:02:13.421061955 (537h33m13.29214499s) [seg_pos={0 0}, upload=471437]: Segment reached optimal threshold of 13 pieces. Dec 28 01:02:13.421382966 (537h33m13.292466001s) [noise=true , address=77.90.31.19:4841 , piece_id=AVZTZGBSQTDACX63JHSIXSO6BAPYX7FIZQZI4XJXMJNKXMTFA4XQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 28 01:02:13.421396673 (537h33m13.292479708s) [noise=true , address=77.90.31.19:4841 , piece_id=AVZTZGBSQTDACX63JHSIXSO6BAPYX7FIZQZI4XJXMJNKXMTFA4XQ, seg_pos={0 0}, upload=471437]: Long tail cancellation due to optimal threshold Dec 28 01:02:13.421400319 (537h33m13.292483374s) [noise=true , address=77.90.31.14:4662 , piece_id=SSZJ22VNDYOSYFIORHW7DF3M6HAPNXOK44NHSRLMU6P3XPFLRBBQ, seg_pos={0 0}, upload=471437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): internal: context canceled Dec 28 01:02:13.421427261 (537h33m13.292510306s) [noise=true , address=77.90.31.14:4662 , piece_id=SSZJ22VNDYOSYFIORHW7DF3M6HAPNXOK44NHSRLMU6P3XPFLRBBQ, seg_pos={0 0}, upload=471437]: Long tail cancellation due to optimal threshold Dec 28 01:02:13.421768710 (537h33m13.292851765s) [noise=true , address=145.79.227.49:4338 , piece_id=5VCWN6CVAJR277DOAZAZ7CNUSVSDEA6WBSKDE5UGJQXK3E2FWBLA, seg_pos={0 0}, upload=471437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 01:02:13.421778860 (537h33m13.292861905s) [noise=true , address=145.79.227.49:4338 , piece_id=5VCWN6CVAJR277DOAZAZ7CNUSVSDEA6WBSKDE5UGJQXK3E2FWBLA, seg_pos={0 0}, upload=471437]: Long tail cancellation due to optimal threshold Dec 28 01:02:13.421851118 (537h33m13.292934163s) [seg_pos={0 0}, upload=471437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:02:13.421969443 (537h33m13.293052488s) [upload=471437]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:02:13.421977768 (537h33m13.293060813s) [upload=471437]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:02:13.714293426 (537h33m13.585376461s) [upload=471437]: Done uploading segments... Dec 28 01:02:13.714312713 (537h33m13.585395748s) [upload=471437]: Upload finished. err: <nil> Duration: 30.002956248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X.] duration: 25.571307172s trace: https://jaeger.dev.storj.io/trace/2aa4dba8475693a8 FAIL (2025-12-28 01:02:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 01:01:12.012002701 (537h32m11.883085736s) [upload=471425]: Starting upload Dec 28 01:01:12.012060662 (537h32m11.883143707s) [upload=471425]: Done starting upload Dec 28 01:01:12.012097753 (537h32m11.883180798s) [upload=471425]: Uploading segments... Dec 28 01:01:12.012354312 (537h32m11.883437347s) [upload=471425]: Got next segment. Inline: false Dec 28 01:01:12.012395260 (537h32m11.883478305s) [upload=471425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:01:12.012399918 (537h32m11.883482953s) [upload=471425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:01:12.259141764 (537h32m12.130224809s) [seg_pos={0 0}, upload=471425]: Begin upload segment... Dec 28 01:01:12.259189626 (537h32m12.130272661s) [seg_pos={0 0}, upload=471425]: Done begin upload segment. Dec 28 01:01:12.259219492 (537h32m12.130302527s) [noise=true , address=45.91.116.8:4048 , piece_id=KLSVO6DFG4SUHBSTUOARWYRQ4INXYEJRHTODQORMTWLJS6F7ACGA, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.259227178 (537h32m12.130310223s) [noise=true , address=145.79.227.24:4581 , piece_id=5O6A3IQZZPLPE22YOZV3NQV4VTY2ICDCI4LMONVPZFCCLTPKZ2WA, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.259972456 (537h32m12.131055501s) [noise=true , address=145.79.227.46:4602 , piece_id=FZBQ6PMC44QLJ5E4K3466VSYRBLTGIUHKVGRSV3BZ3ZPPUUTXENQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.260544946 (537h32m12.131628001s) [noise=true , address=77.90.31.11:4555 , piece_id=YN4MPNDAVE4U7KBFAEP34IPVKMO7RUQ4MCZUJLTPCUC2RCN7WU3Q, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.260987469 (537h32m12.132070514s) [noise=true , address=145.79.227.42:4122 , piece_id=STZUYJSYYWJD3OQUI6I57PW723ZMAE46LVVLTPP2O7M4Z3ASGGBQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.261441684 (537h32m12.132524729s) [noise=true , address=145.79.227.14:4838 , piece_id=ATXMBZ2I6HGPQ6XNHDWZPKDIXP2GZ7REGABISV32M2S3O3OXMDAQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.261962135 (537h32m12.13304518s) [noise=true , address=45.91.116.7:4011 , piece_id=7XBWJQPMPB4W3XLBLREATJFN3ZMBIQFNFOJ2JY3FPRUKFO6KICBA, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.262535247 (537h32m12.133618282s) [noise=true , address=145.79.227.19:4079 , piece_id=UNGTWMPCXH23PWNLDEOQJHWUUIDNBJYLLLPJ7QYSJHU22S7XAZPQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.263098148 (537h32m12.134181193s) [noise=true , address=145.79.227.15:4873 , piece_id=DNZ5XJP3OXYXQOQJRPZLAKE2XA7J54EGIS6VZ33HA5GJLP2SNEUA, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.263107917 (537h32m12.134190962s) [noise=true , address=45.91.116.12:4227 , piece_id=JJ4WKS6SQZKC774ULVSDD6A4OPOY4K5NSA477CJWS6WPGD6W2BDQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.263604001 (537h32m12.134687046s) [noise=true , address=45.91.116.11:4119 , piece_id=5CF676N65FHBZLHJJ6JG6FTFLSCLOATF2HZMJMTMX22IUFJFJLPA, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.264095367 (537h32m12.135178412s) [noise=true , address=45.91.116.14:4190 , piece_id=BNL6OBPYMT5DPIXGP7QANOBCYG23SWRGRHJYDLDRGE6HPKM62VMQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.265461769 (537h32m12.136544824s) [noise=true , address=45.91.116.4:4937 , piece_id=VSJSIMIR6C7IOI7ZRE5WOL2Z75K5ORDBRAAUQWHY5EFCU3ZCQ4KQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.265881338 (537h32m12.136964373s) [noise=true , address=77.90.31.16:4735 , piece_id=AARFM7ENE4XVPF44BVZOSY5EI7MW3OVTHNCPMJRIQX2YVJMJU23A, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.267626461 (537h32m12.138709506s) [noise=true , address=45.91.116.13:4155 , piece_id=G4IAKOIQPOMQBPGGGQTUQPWNIZADUY5LFHYCETQCQQPA2JZ75FMQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.269099135 (537h32m12.14018217s) [noise=true , address=145.79.227.37:4567 , piece_id=42EOAA3IKDXQ4D5Y5TU6NJSRXENS22DICARXDZEPIMHTRLKIAH5A, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:13.984269862 (537h32m13.855352897s) [upload=471425]: Next returned nil segment Dec 28 01:01:13.984285993 (537h32m13.855369038s) [upload=471425]: Waiting for error group managing segments... Dec 28 01:01:14.269205155 (537h32m14.140288201s) [noise=true , address=145.79.227.46:4602 , piece_id=FZBQ6PMC44QLJ5E4K3466VSYRBLTGIUHKVGRSV3BZ3ZPPUUTXENQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.272063658 (537h32m14.143146703s) [noise=true , address=145.79.227.42:4122 , piece_id=STZUYJSYYWJD3OQUI6I57PW723ZMAE46LVVLTPP2O7M4Z3ASGGBQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.280752232 (537h32m14.151835277s) [noise=true , address=145.79.227.19:4079 , piece_id=UNGTWMPCXH23PWNLDEOQJHWUUIDNBJYLLLPJ7QYSJHU22S7XAZPQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.283300314 (537h32m14.154383359s) [noise=true , address=145.79.227.37:4567 , piece_id=42EOAA3IKDXQ4D5Y5TU6NJSRXENS22DICARXDZEPIMHTRLKIAH5A, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.290644479 (537h32m14.161727534s) [noise=true , address=145.79.227.24:4581 , piece_id=5O6A3IQZZPLPE22YOZV3NQV4VTY2ICDCI4LMONVPZFCCLTPKZ2WA, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.299554665 (537h32m14.17063771s) [noise=true , address=145.79.227.15:4873 , piece_id=DNZ5XJP3OXYXQOQJRPZLAKE2XA7J54EGIS6VZ33HA5GJLP2SNEUA, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.312717210 (537h32m14.183800255s) [noise=true , address=145.79.227.14:4838 , piece_id=ATXMBZ2I6HGPQ6XNHDWZPKDIXP2GZ7REGABISV32M2S3O3OXMDAQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.481528093 (537h32m14.352611138s) [noise=true , address=77.90.31.11:4555 , piece_id=YN4MPNDAVE4U7KBFAEP34IPVKMO7RUQ4MCZUJLTPCUC2RCN7WU3Q, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.508429127 (537h32m14.379512172s) [noise=true , address=45.91.116.13:4155 , piece_id=G4IAKOIQPOMQBPGGGQTUQPWNIZADUY5LFHYCETQCQQPA2JZ75FMQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.514614865 (537h32m14.3856979s) [noise=true , address=45.91.116.14:4190 , piece_id=BNL6OBPYMT5DPIXGP7QANOBCYG23SWRGRHJYDLDRGE6HPKM62VMQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.524352227 (537h32m14.395435272s) [noise=true , address=45.91.116.12:4227 , piece_id=JJ4WKS6SQZKC774ULVSDD6A4OPOY4K5NSA477CJWS6WPGD6W2BDQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.535732878 (537h32m14.406815933s) [noise=true , address=45.91.116.7:4011 , piece_id=7XBWJQPMPB4W3XLBLREATJFN3ZMBIQFNFOJ2JY3FPRUKFO6KICBA, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.540272642 (537h32m14.411355687s) [noise=true , address=45.91.116.8:4048 , piece_id=KLSVO6DFG4SUHBSTUOARWYRQ4INXYEJRHTODQORMTWLJS6F7ACGA, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.540296107 (537h32m14.411379142s) [seg_pos={0 0}, upload=471425]: Segment reached optimal threshold of 13 pieces. Dec 28 01:01:14.540531155 (537h32m14.4116142s) [noise=true , address=77.90.31.16:4735 , piece_id=AARFM7ENE4XVPF44BVZOSY5EI7MW3OVTHNCPMJRIQX2YVJMJU23A, seg_pos={0 0}, upload=471425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 01:01:14.540541364 (537h32m14.411624409s) [noise=true , address=77.90.31.16:4735 , piece_id=AARFM7ENE4XVPF44BVZOSY5EI7MW3OVTHNCPMJRIQX2YVJMJU23A, seg_pos={0 0}, upload=471425]: Long tail cancellation due to optimal threshold Dec 28 01:01:14.540707921 (537h32m14.411790956s) [noise=true , address=45.91.116.4:4937 , piece_id=VSJSIMIR6C7IOI7ZRE5WOL2Z75K5ORDBRAAUQWHY5EFCU3ZCQ4KQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 01:01:14.540724172 (537h32m14.411807207s) [noise=true , address=45.91.116.4:4937 , piece_id=VSJSIMIR6C7IOI7ZRE5WOL2Z75K5ORDBRAAUQWHY5EFCU3ZCQ4KQ, seg_pos={0 0}, upload=471425]: Long tail cancellation due to optimal threshold Dec 28 01:01:14.540742106 (537h32m14.411825151s) [noise=true , address=45.91.116.11:4119 , piece_id=5CF676N65FHBZLHJJ6JG6FTFLSCLOATF2HZMJMTMX22IUFJFJLPA, seg_pos={0 0}, upload=471425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 01:01:14.540752055 (537h32m14.41183513s) [noise=true , address=45.91.116.11:4119 , piece_id=5CF676N65FHBZLHJJ6JG6FTFLSCLOATF2HZMJMTMX22IUFJFJLPA, seg_pos={0 0}, upload=471425]: Long tail cancellation due to optimal threshold Dec 28 01:01:14.540782523 (537h32m14.411865568s) [seg_pos={0 0}, upload=471425]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:01:14.540887122 (537h32m14.411970167s) [upload=471425]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:01:14.540894967 (537h32m14.411978022s) [upload=471425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:01:14.841122671 (537h32m14.712205716s) [upload=471425]: Done uploading segments... Dec 28 01:01:14.841147347 (537h32m14.712230422s) [upload=471425]: Upload finished. err: <nil> Duration: 25.571326568s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 17.00986913s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 01:01:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 17.009898275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.X] duration: 30.002229571s trace: https://jaeger.dev.storj.io/trace/2aa4dba8475693a8 FAIL (2025-12-28 01:01:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 01:01:12.012002701 (537h32m11.883085736s) [upload=471425]: Starting upload Dec 28 01:01:12.012060662 (537h32m11.883143707s) [upload=471425]: Done starting upload Dec 28 01:01:12.012097753 (537h32m11.883180798s) [upload=471425]: Uploading segments... Dec 28 01:01:12.012354312 (537h32m11.883437347s) [upload=471425]: Got next segment. Inline: false Dec 28 01:01:12.012395260 (537h32m11.883478305s) [upload=471425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 01:01:12.012399918 (537h32m11.883482953s) [upload=471425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 01:01:12.259141764 (537h32m12.130224809s) [seg_pos={0 0}, upload=471425]: Begin upload segment... Dec 28 01:01:12.259189626 (537h32m12.130272661s) [seg_pos={0 0}, upload=471425]: Done begin upload segment. Dec 28 01:01:12.259219492 (537h32m12.130302527s) [noise=true , address=45.91.116.8:4048 , piece_id=KLSVO6DFG4SUHBSTUOARWYRQ4INXYEJRHTODQORMTWLJS6F7ACGA, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.259227178 (537h32m12.130310223s) [noise=true , address=145.79.227.24:4581 , piece_id=5O6A3IQZZPLPE22YOZV3NQV4VTY2ICDCI4LMONVPZFCCLTPKZ2WA, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.259972456 (537h32m12.131055501s) [noise=true , address=145.79.227.46:4602 , piece_id=FZBQ6PMC44QLJ5E4K3466VSYRBLTGIUHKVGRSV3BZ3ZPPUUTXENQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.260544946 (537h32m12.131628001s) [noise=true , address=77.90.31.11:4555 , piece_id=YN4MPNDAVE4U7KBFAEP34IPVKMO7RUQ4MCZUJLTPCUC2RCN7WU3Q, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.260987469 (537h32m12.132070514s) [noise=true , address=145.79.227.42:4122 , piece_id=STZUYJSYYWJD3OQUI6I57PW723ZMAE46LVVLTPP2O7M4Z3ASGGBQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.261441684 (537h32m12.132524729s) [noise=true , address=145.79.227.14:4838 , piece_id=ATXMBZ2I6HGPQ6XNHDWZPKDIXP2GZ7REGABISV32M2S3O3OXMDAQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.261962135 (537h32m12.13304518s) [noise=true , address=45.91.116.7:4011 , piece_id=7XBWJQPMPB4W3XLBLREATJFN3ZMBIQFNFOJ2JY3FPRUKFO6KICBA, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.262535247 (537h32m12.133618282s) [noise=true , address=145.79.227.19:4079 , piece_id=UNGTWMPCXH23PWNLDEOQJHWUUIDNBJYLLLPJ7QYSJHU22S7XAZPQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.263098148 (537h32m12.134181193s) [noise=true , address=145.79.227.15:4873 , piece_id=DNZ5XJP3OXYXQOQJRPZLAKE2XA7J54EGIS6VZ33HA5GJLP2SNEUA, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.263107917 (537h32m12.134190962s) [noise=true , address=45.91.116.12:4227 , piece_id=JJ4WKS6SQZKC774ULVSDD6A4OPOY4K5NSA477CJWS6WPGD6W2BDQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.263604001 (537h32m12.134687046s) [noise=true , address=45.91.116.11:4119 , piece_id=5CF676N65FHBZLHJJ6JG6FTFLSCLOATF2HZMJMTMX22IUFJFJLPA, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.264095367 (537h32m12.135178412s) [noise=true , address=45.91.116.14:4190 , piece_id=BNL6OBPYMT5DPIXGP7QANOBCYG23SWRGRHJYDLDRGE6HPKM62VMQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.265461769 (537h32m12.136544824s) [noise=true , address=45.91.116.4:4937 , piece_id=VSJSIMIR6C7IOI7ZRE5WOL2Z75K5ORDBRAAUQWHY5EFCU3ZCQ4KQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.265881338 (537h32m12.136964373s) [noise=true , address=77.90.31.16:4735 , piece_id=AARFM7ENE4XVPF44BVZOSY5EI7MW3OVTHNCPMJRIQX2YVJMJU23A, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.267626461 (537h32m12.138709506s) [noise=true , address=45.91.116.13:4155 , piece_id=G4IAKOIQPOMQBPGGGQTUQPWNIZADUY5LFHYCETQCQQPA2JZ75FMQ, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:12.269099135 (537h32m12.14018217s) [noise=true , address=145.79.227.37:4567 , piece_id=42EOAA3IKDXQ4D5Y5TU6NJSRXENS22DICARXDZEPIMHTRLKIAH5A, seg_pos={0 0}, upload=471425]: Uploading piece... Dec 28 01:01:13.984269862 (537h32m13.855352897s) [upload=471425]: Next returned nil segment Dec 28 01:01:13.984285993 (537h32m13.855369038s) [upload=471425]: Waiting for error group managing segments... Dec 28 01:01:14.269205155 (537h32m14.140288201s) [noise=true , address=145.79.227.46:4602 , piece_id=FZBQ6PMC44QLJ5E4K3466VSYRBLTGIUHKVGRSV3BZ3ZPPUUTXENQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.272063658 (537h32m14.143146703s) [noise=true , address=145.79.227.42:4122 , piece_id=STZUYJSYYWJD3OQUI6I57PW723ZMAE46LVVLTPP2O7M4Z3ASGGBQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.280752232 (537h32m14.151835277s) [noise=true , address=145.79.227.19:4079 , piece_id=UNGTWMPCXH23PWNLDEOQJHWUUIDNBJYLLLPJ7QYSJHU22S7XAZPQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.283300314 (537h32m14.154383359s) [noise=true , address=145.79.227.37:4567 , piece_id=42EOAA3IKDXQ4D5Y5TU6NJSRXENS22DICARXDZEPIMHTRLKIAH5A, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.290644479 (537h32m14.161727534s) [noise=true , address=145.79.227.24:4581 , piece_id=5O6A3IQZZPLPE22YOZV3NQV4VTY2ICDCI4LMONVPZFCCLTPKZ2WA, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.299554665 (537h32m14.17063771s) [noise=true , address=145.79.227.15:4873 , piece_id=DNZ5XJP3OXYXQOQJRPZLAKE2XA7J54EGIS6VZ33HA5GJLP2SNEUA, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.312717210 (537h32m14.183800255s) [noise=true , address=145.79.227.14:4838 , piece_id=ATXMBZ2I6HGPQ6XNHDWZPKDIXP2GZ7REGABISV32M2S3O3OXMDAQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.481528093 (537h32m14.352611138s) [noise=true , address=77.90.31.11:4555 , piece_id=YN4MPNDAVE4U7KBFAEP34IPVKMO7RUQ4MCZUJLTPCUC2RCN7WU3Q, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.508429127 (537h32m14.379512172s) [noise=true , address=45.91.116.13:4155 , piece_id=G4IAKOIQPOMQBPGGGQTUQPWNIZADUY5LFHYCETQCQQPA2JZ75FMQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.514614865 (537h32m14.3856979s) [noise=true , address=45.91.116.14:4190 , piece_id=BNL6OBPYMT5DPIXGP7QANOBCYG23SWRGRHJYDLDRGE6HPKM62VMQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.524352227 (537h32m14.395435272s) [noise=true , address=45.91.116.12:4227 , piece_id=JJ4WKS6SQZKC774ULVSDD6A4OPOY4K5NSA477CJWS6WPGD6W2BDQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.535732878 (537h32m14.406815933s) [noise=true , address=45.91.116.7:4011 , piece_id=7XBWJQPMPB4W3XLBLREATJFN3ZMBIQFNFOJ2JY3FPRUKFO6KICBA, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.540272642 (537h32m14.411355687s) [noise=true , address=45.91.116.8:4048 , piece_id=KLSVO6DFG4SUHBSTUOARWYRQ4INXYEJRHTODQORMTWLJS6F7ACGA, seg_pos={0 0}, upload=471425]: Done uploading piece. err: <nil> Dec 28 01:01:14.540296107 (537h32m14.411379142s) [seg_pos={0 0}, upload=471425]: Segment reached optimal threshold of 13 pieces. Dec 28 01:01:14.540531155 (537h32m14.4116142s) [noise=true , address=77.90.31.16:4735 , piece_id=AARFM7ENE4XVPF44BVZOSY5EI7MW3OVTHNCPMJRIQX2YVJMJU23A, seg_pos={0 0}, upload=471425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 01:01:14.540541364 (537h32m14.411624409s) [noise=true , address=77.90.31.16:4735 , piece_id=AARFM7ENE4XVPF44BVZOSY5EI7MW3OVTHNCPMJRIQX2YVJMJU23A, seg_pos={0 0}, upload=471425]: Long tail cancellation due to optimal threshold Dec 28 01:01:14.540707921 (537h32m14.411790956s) [noise=true , address=45.91.116.4:4937 , piece_id=VSJSIMIR6C7IOI7ZRE5WOL2Z75K5ORDBRAAUQWHY5EFCU3ZCQ4KQ, seg_pos={0 0}, upload=471425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 28 01:01:14.540724172 (537h32m14.411807207s) [noise=true , address=45.91.116.4:4937 , piece_id=VSJSIMIR6C7IOI7ZRE5WOL2Z75K5ORDBRAAUQWHY5EFCU3ZCQ4KQ, seg_pos={0 0}, upload=471425]: Long tail cancellation due to optimal threshold Dec 28 01:01:14.540742106 (537h32m14.411825151s) [noise=true , address=45.91.116.11:4119 , piece_id=5CF676N65FHBZLHJJ6JG6FTFLSCLOATF2HZMJMTMX22IUFJFJLPA, seg_pos={0 0}, upload=471425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 01:01:14.540752055 (537h32m14.41183513s) [noise=true , address=45.91.116.11:4119 , piece_id=5CF676N65FHBZLHJJ6JG6FTFLSCLOATF2HZMJMTMX22IUFJFJLPA, seg_pos={0 0}, upload=471425]: Long tail cancellation due to optimal threshold Dec 28 01:01:14.540782523 (537h32m14.411865568s) [seg_pos={0 0}, upload=471425]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 01:01:14.540887122 (537h32m14.411970167s) [upload=471425]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 01:01:14.540894967 (537h32m14.411978022s) [upload=471425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 01:01:14.841122671 (537h32m14.712205716s) [upload=471425]: Done uploading segments... Dec 28 01:01:14.841147347 (537h32m14.712230422s) [upload=471425]: Upload finished. err: <nil> Duration: 30.002254167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 15.413909508s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 01:00:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 15.413926261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 27.366960916s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 01:00:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 27.366994371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X.] duration: 6.023539508s trace: https://jaeger.dev.storj.io/trace/20bba755fe350552 FAIL (2025-12-28 01:00:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: N23VVNXSEEDZRVJ3Q7CUDGULNTCHKEUQ6Q2Y5Z5VTZAVO726QDMQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: UJVBGUFDZJTBQXLKGENCWKESLMD56DDEN3IX52E6L7JXPC2BSJUA) context deadline exceeded ======LOGS====== Dec 28 00:58:55.722719668 (537h29m55.593802703s) [upload=471395]: Starting upload Dec 28 00:58:55.722785703 (537h29m55.593868748s) [upload=471395]: Done starting upload Dec 28 00:58:55.722885103 (537h29m55.593968148s) [upload=471395]: Uploading segments... Dec 28 00:58:55.722966818 (537h29m55.594049873s) [upload=471395]: Got next segment. Inline: false Dec 28 00:58:55.723028255 (537h29m55.59411129s) [upload=471395]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:58:55.723035018 (537h29m55.594118063s) [upload=471395]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:58:55.965303116 (537h29m55.836386151s) [seg_pos={0 0}, upload=471395]: Begin upload segment... Dec 28 00:58:55.965365044 (537h29m55.836448079s) [seg_pos={0 0}, upload=471395]: Done begin upload segment. Dec 28 00:58:55.965445747 (537h29m55.836528792s) [noise=true , address=145.79.227.19:4078 , piece_id=AISMXFZ55LTA3HB4TRXS34EMBSVUAZPVJCGOG3NN63RLJDQSEMUA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.965469252 (537h29m55.836552287s) [noise=true , address=45.91.116.5:4298 , piece_id=N3DHCKLSNB3V2WQA3LGRQU4NJ6FVHRSQR2Q7CEVPJXSRGDDM73WQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.966051892 (537h29m55.837134937s) [noise=true , address=145.79.227.43:4159 , piece_id=UQBRTTTRHKZ7Y2P7P4FJRRJD4M52HHUH4SZRB2BQGUIVSLLWAOTA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.966609814 (537h29m55.837692859s) [noise=true , address=145.79.227.44:4194 , piece_id=SI52P53FVLIGM5SFJXYZ2FF4MY6B3B72TDPAZMDHX4ITMPOINJBA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.966625414 (537h29m55.837708449s) [noise=true , address=145.79.227.36:4519 , piece_id=AWM3EN4D25MT3ROCIVLSIGD6BSA6D32QEDXBCLWWEKGSXZ7F5WKA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.967001640 (537h29m55.838084675s) [noise=true , address=77.90.31.19:4844 , piece_id=MAIH3I2GWJOWJXWJODDFSAVU4YJ3JYNS6YMSFOH2ILJ7Y4XOE7BQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.967027700 (537h29m55.838110735s) [noise=true , address=145.79.227.35:4483 , piece_id=HGNYUGVGBSCPNJC426RNONFYYMQTIAANEVIVDNNCNFSIVNS3SM6A, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.967355945 (537h29m55.83843898s) [noise=true , address=45.91.116.11:4119 , piece_id=UJVBGUFDZJTBQXLKGENCWKESLMD56DDEN3IX52E6L7JXPC2BSJUA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.967684952 (537h29m55.838767997s) [noise=true , address=77.90.31.11:4554 , piece_id=AITHH5W3XQ6RZBVKHIIHBN74EXBV3WJZLVZZXMFJYOI5OLWLX5KA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.968068813 (537h29m55.839151848s) [noise=true , address=145.79.227.42:4121 , piece_id=YMTBHP3MWZKYMIRLW3IGNHPVLE6L25UMHQEQJODFXXL4FF63WABA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.968414952 (537h29m55.839497997s) [noise=true , address=145.79.227.4:4187 , piece_id=SPHI47WKDBLTTQBHEOGM2SHZUXK7VAVUGYYDN6UX4NRZNT7EGJOQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.968811376 (537h29m55.839894411s) [noise=true , address=145.79.227.11:4729 , piece_id=4LH4BOU7ZAY3KLVK26UOMIVA4FNZPODYXWINGSKFFQXF5GSPTDHQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.969185349 (537h29m55.840268394s) [noise=true , address=145.79.227.16:4910 , piece_id=2F4YVQHOJQH462LY5K3VYMCD2SMJZ2ZRLC3MGIK44MGU6BKKBT5A, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.969589317 (537h29m55.840672352s) [noise=true , address=45.91.116.9:4264 , piece_id=N23VVNXSEEDZRVJ3Q7CUDGULNTCHKEUQ6Q2Y5Z5VTZAVO726QDMQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.970084381 (537h29m55.841167426s) [noise=true , address=145.79.227.14:4839 , piece_id=IWLK6OQVAGT4GCGTRDVDN4FHAJWBKU7SAPL532VIPMWMV6X7MQZQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.972853964 (537h29m55.843936999s) [noise=true , address=145.79.227.38:4979 , piece_id=CLUEA5BNTLGIALEIKB2LZL7BU3A2VLTRXEPI54Q6L5IVN5424ASQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:57.687785957 (537h29m57.558868992s) [upload=471395]: Next returned nil segment Dec 28 00:58:57.687802458 (537h29m57.558885493s) [upload=471395]: Waiting for error group managing segments... Dec 28 00:58:57.968089500 (537h29m57.839172545s) [noise=true , address=145.79.227.16:4910 , piece_id=2F4YVQHOJQH462LY5K3VYMCD2SMJZ2ZRLC3MGIK44MGU6BKKBT5A, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.970867029 (537h29m57.841950074s) [noise=true , address=145.79.227.38:4979 , piece_id=CLUEA5BNTLGIALEIKB2LZL7BU3A2VLTRXEPI54Q6L5IVN5424ASQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.975691807 (537h29m57.846774842s) [noise=true , address=145.79.227.42:4121 , piece_id=YMTBHP3MWZKYMIRLW3IGNHPVLE6L25UMHQEQJODFXXL4FF63WABA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.985546261 (537h29m57.856629296s) [noise=true , address=145.79.227.11:4729 , piece_id=4LH4BOU7ZAY3KLVK26UOMIVA4FNZPODYXWINGSKFFQXF5GSPTDHQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.987910073 (537h29m57.858993108s) [noise=true , address=145.79.227.4:4187 , piece_id=SPHI47WKDBLTTQBHEOGM2SHZUXK7VAVUGYYDN6UX4NRZNT7EGJOQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.993902733 (537h29m57.864985779s) [noise=true , address=145.79.227.36:4519 , piece_id=AWM3EN4D25MT3ROCIVLSIGD6BSA6D32QEDXBCLWWEKGSXZ7F5WKA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.998499136 (537h29m57.869582171s) [noise=true , address=145.79.227.14:4839 , piece_id=IWLK6OQVAGT4GCGTRDVDN4FHAJWBKU7SAPL532VIPMWMV6X7MQZQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.189882541 (537h29m58.060965576s) [noise=true , address=77.90.31.11:4554 , piece_id=AITHH5W3XQ6RZBVKHIIHBN74EXBV3WJZLVZZXMFJYOI5OLWLX5KA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.203881477 (537h29m58.074964522s) [noise=true , address=77.90.31.19:4844 , piece_id=MAIH3I2GWJOWJXWJODDFSAVU4YJ3JYNS6YMSFOH2ILJ7Y4XOE7BQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.281295005 (537h29m58.15237805s) [noise=true , address=45.91.116.5:4298 , piece_id=N3DHCKLSNB3V2WQA3LGRQU4NJ6FVHRSQR2Q7CEVPJXSRGDDM73WQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.336613391 (537h29m58.207696426s) [noise=true , address=145.79.227.19:4078 , piece_id=AISMXFZ55LTA3HB4TRXS34EMBSVUAZPVJCGOG3NN63RLJDQSEMUA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.347599110 (537h29m58.218682155s) [noise=true , address=45.91.116.9:4264 , piece_id=N23VVNXSEEDZRVJ3Q7CUDGULNTCHKEUQ6Q2Y5Z5VTZAVO726QDMQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.554926685 (537h29m58.42600972s) [noise=true , address=45.91.116.11:4119 , piece_id=UJVBGUFDZJTBQXLKGENCWKESLMD56DDEN3IX52E6L7JXPC2BSJUA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.554943968 (537h29m58.426027003s) [seg_pos={0 0}, upload=471395]: Segment reached optimal threshold of 13 pieces. Dec 28 00:58:58.555120965 (537h29m58.426204s) [noise=true , address=145.79.227.35:4483 , piece_id=HGNYUGVGBSCPNJC426RNONFYYMQTIAANEVIVDNNCNFSIVNS3SM6A, seg_pos={0 0}, upload=471395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 00:58:58.555131755 (537h29m58.42621479s) [noise=true , address=145.79.227.35:4483 , piece_id=HGNYUGVGBSCPNJC426RNONFYYMQTIAANEVIVDNNCNFSIVNS3SM6A, seg_pos={0 0}, upload=471395]: Long tail cancellation due to optimal threshold Dec 28 00:58:58.555231676 (537h29m58.426314711s) [noise=true , address=145.79.227.43:4159 , piece_id=UQBRTTTRHKZ7Y2P7P4FJRRJD4M52HHUH4SZRB2BQGUIVSLLWAOTA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 00:58:58.555237547 (537h29m58.426320582s) [noise=true , address=145.79.227.43:4159 , piece_id=UQBRTTTRHKZ7Y2P7P4FJRRJD4M52HHUH4SZRB2BQGUIVSLLWAOTA, seg_pos={0 0}, upload=471395]: Long tail cancellation due to optimal threshold Dec 28 00:58:58.555265921 (537h29m58.426348956s) [noise=true , address=145.79.227.44:4194 , piece_id=SI52P53FVLIGM5SFJXYZ2FF4MY6B3B72TDPAZMDHX4ITMPOINJBA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 00:58:58.555271050 (537h29m58.426354085s) [noise=true , address=145.79.227.44:4194 , piece_id=SI52P53FVLIGM5SFJXYZ2FF4MY6B3B72TDPAZMDHX4ITMPOINJBA, seg_pos={0 0}, upload=471395]: Long tail cancellation due to optimal threshold Dec 28 00:58:58.555285649 (537h29m58.426368684s) [seg_pos={0 0}, upload=471395]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:58:58.555354009 (537h29m58.426437044s) [upload=471395]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:58:58.555358517 (537h29m58.426441562s) [upload=471395]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:58:58.843284555 (537h29m58.71436759s) [upload=471395]: Done uploading segments... Dec 28 00:58:58.843307820 (537h29m58.714390855s) [upload=471395]: Upload finished. err: <nil> Duration: 6.023569336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 26.236791038s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 01:00:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 26.236824833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....X] duration: 30.002967937s trace: https://jaeger.dev.storj.io/trace/20bba755fe350552 FAIL (2025-12-28 00:59:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: N23VVNXSEEDZRVJ3Q7CUDGULNTCHKEUQ6Q2Y5Z5VTZAVO726QDMQ) context deadline exceeded; piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: UJVBGUFDZJTBQXLKGENCWKESLMD56DDEN3IX52E6L7JXPC2BSJUA) context deadline exceeded ======LOGS====== Dec 28 00:58:55.722719668 (537h29m55.593802703s) [upload=471395]: Starting upload Dec 28 00:58:55.722785703 (537h29m55.593868748s) [upload=471395]: Done starting upload Dec 28 00:58:55.722885103 (537h29m55.593968148s) [upload=471395]: Uploading segments... Dec 28 00:58:55.722966818 (537h29m55.594049873s) [upload=471395]: Got next segment. Inline: false Dec 28 00:58:55.723028255 (537h29m55.59411129s) [upload=471395]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:58:55.723035018 (537h29m55.594118063s) [upload=471395]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:58:55.965303116 (537h29m55.836386151s) [seg_pos={0 0}, upload=471395]: Begin upload segment... Dec 28 00:58:55.965365044 (537h29m55.836448079s) [seg_pos={0 0}, upload=471395]: Done begin upload segment. Dec 28 00:58:55.965445747 (537h29m55.836528792s) [noise=true , address=145.79.227.19:4078 , piece_id=AISMXFZ55LTA3HB4TRXS34EMBSVUAZPVJCGOG3NN63RLJDQSEMUA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.965469252 (537h29m55.836552287s) [noise=true , address=45.91.116.5:4298 , piece_id=N3DHCKLSNB3V2WQA3LGRQU4NJ6FVHRSQR2Q7CEVPJXSRGDDM73WQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.966051892 (537h29m55.837134937s) [noise=true , address=145.79.227.43:4159 , piece_id=UQBRTTTRHKZ7Y2P7P4FJRRJD4M52HHUH4SZRB2BQGUIVSLLWAOTA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.966609814 (537h29m55.837692859s) [noise=true , address=145.79.227.44:4194 , piece_id=SI52P53FVLIGM5SFJXYZ2FF4MY6B3B72TDPAZMDHX4ITMPOINJBA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.966625414 (537h29m55.837708449s) [noise=true , address=145.79.227.36:4519 , piece_id=AWM3EN4D25MT3ROCIVLSIGD6BSA6D32QEDXBCLWWEKGSXZ7F5WKA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.967001640 (537h29m55.838084675s) [noise=true , address=77.90.31.19:4844 , piece_id=MAIH3I2GWJOWJXWJODDFSAVU4YJ3JYNS6YMSFOH2ILJ7Y4XOE7BQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.967027700 (537h29m55.838110735s) [noise=true , address=145.79.227.35:4483 , piece_id=HGNYUGVGBSCPNJC426RNONFYYMQTIAANEVIVDNNCNFSIVNS3SM6A, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.967355945 (537h29m55.83843898s) [noise=true , address=45.91.116.11:4119 , piece_id=UJVBGUFDZJTBQXLKGENCWKESLMD56DDEN3IX52E6L7JXPC2BSJUA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.967684952 (537h29m55.838767997s) [noise=true , address=77.90.31.11:4554 , piece_id=AITHH5W3XQ6RZBVKHIIHBN74EXBV3WJZLVZZXMFJYOI5OLWLX5KA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.968068813 (537h29m55.839151848s) [noise=true , address=145.79.227.42:4121 , piece_id=YMTBHP3MWZKYMIRLW3IGNHPVLE6L25UMHQEQJODFXXL4FF63WABA, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.968414952 (537h29m55.839497997s) [noise=true , address=145.79.227.4:4187 , piece_id=SPHI47WKDBLTTQBHEOGM2SHZUXK7VAVUGYYDN6UX4NRZNT7EGJOQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.968811376 (537h29m55.839894411s) [noise=true , address=145.79.227.11:4729 , piece_id=4LH4BOU7ZAY3KLVK26UOMIVA4FNZPODYXWINGSKFFQXF5GSPTDHQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.969185349 (537h29m55.840268394s) [noise=true , address=145.79.227.16:4910 , piece_id=2F4YVQHOJQH462LY5K3VYMCD2SMJZ2ZRLC3MGIK44MGU6BKKBT5A, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.969589317 (537h29m55.840672352s) [noise=true , address=45.91.116.9:4264 , piece_id=N23VVNXSEEDZRVJ3Q7CUDGULNTCHKEUQ6Q2Y5Z5VTZAVO726QDMQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.970084381 (537h29m55.841167426s) [noise=true , address=145.79.227.14:4839 , piece_id=IWLK6OQVAGT4GCGTRDVDN4FHAJWBKU7SAPL532VIPMWMV6X7MQZQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:55.972853964 (537h29m55.843936999s) [noise=true , address=145.79.227.38:4979 , piece_id=CLUEA5BNTLGIALEIKB2LZL7BU3A2VLTRXEPI54Q6L5IVN5424ASQ, seg_pos={0 0}, upload=471395]: Uploading piece... Dec 28 00:58:57.687785957 (537h29m57.558868992s) [upload=471395]: Next returned nil segment Dec 28 00:58:57.687802458 (537h29m57.558885493s) [upload=471395]: Waiting for error group managing segments... Dec 28 00:58:57.968089500 (537h29m57.839172545s) [noise=true , address=145.79.227.16:4910 , piece_id=2F4YVQHOJQH462LY5K3VYMCD2SMJZ2ZRLC3MGIK44MGU6BKKBT5A, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.970867029 (537h29m57.841950074s) [noise=true , address=145.79.227.38:4979 , piece_id=CLUEA5BNTLGIALEIKB2LZL7BU3A2VLTRXEPI54Q6L5IVN5424ASQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.975691807 (537h29m57.846774842s) [noise=true , address=145.79.227.42:4121 , piece_id=YMTBHP3MWZKYMIRLW3IGNHPVLE6L25UMHQEQJODFXXL4FF63WABA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.985546261 (537h29m57.856629296s) [noise=true , address=145.79.227.11:4729 , piece_id=4LH4BOU7ZAY3KLVK26UOMIVA4FNZPODYXWINGSKFFQXF5GSPTDHQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.987910073 (537h29m57.858993108s) [noise=true , address=145.79.227.4:4187 , piece_id=SPHI47WKDBLTTQBHEOGM2SHZUXK7VAVUGYYDN6UX4NRZNT7EGJOQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.993902733 (537h29m57.864985779s) [noise=true , address=145.79.227.36:4519 , piece_id=AWM3EN4D25MT3ROCIVLSIGD6BSA6D32QEDXBCLWWEKGSXZ7F5WKA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:57.998499136 (537h29m57.869582171s) [noise=true , address=145.79.227.14:4839 , piece_id=IWLK6OQVAGT4GCGTRDVDN4FHAJWBKU7SAPL532VIPMWMV6X7MQZQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.189882541 (537h29m58.060965576s) [noise=true , address=77.90.31.11:4554 , piece_id=AITHH5W3XQ6RZBVKHIIHBN74EXBV3WJZLVZZXMFJYOI5OLWLX5KA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.203881477 (537h29m58.074964522s) [noise=true , address=77.90.31.19:4844 , piece_id=MAIH3I2GWJOWJXWJODDFSAVU4YJ3JYNS6YMSFOH2ILJ7Y4XOE7BQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.281295005 (537h29m58.15237805s) [noise=true , address=45.91.116.5:4298 , piece_id=N3DHCKLSNB3V2WQA3LGRQU4NJ6FVHRSQR2Q7CEVPJXSRGDDM73WQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.336613391 (537h29m58.207696426s) [noise=true , address=145.79.227.19:4078 , piece_id=AISMXFZ55LTA3HB4TRXS34EMBSVUAZPVJCGOG3NN63RLJDQSEMUA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.347599110 (537h29m58.218682155s) [noise=true , address=45.91.116.9:4264 , piece_id=N23VVNXSEEDZRVJ3Q7CUDGULNTCHKEUQ6Q2Y5Z5VTZAVO726QDMQ, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.554926685 (537h29m58.42600972s) [noise=true , address=45.91.116.11:4119 , piece_id=UJVBGUFDZJTBQXLKGENCWKESLMD56DDEN3IX52E6L7JXPC2BSJUA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: <nil> Dec 28 00:58:58.554943968 (537h29m58.426027003s) [seg_pos={0 0}, upload=471395]: Segment reached optimal threshold of 13 pieces. Dec 28 00:58:58.555120965 (537h29m58.426204s) [noise=true , address=145.79.227.35:4483 , piece_id=HGNYUGVGBSCPNJC426RNONFYYMQTIAANEVIVDNNCNFSIVNS3SM6A, seg_pos={0 0}, upload=471395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 00:58:58.555131755 (537h29m58.42621479s) [noise=true , address=145.79.227.35:4483 , piece_id=HGNYUGVGBSCPNJC426RNONFYYMQTIAANEVIVDNNCNFSIVNS3SM6A, seg_pos={0 0}, upload=471395]: Long tail cancellation due to optimal threshold Dec 28 00:58:58.555231676 (537h29m58.426314711s) [noise=true , address=145.79.227.43:4159 , piece_id=UQBRTTTRHKZ7Y2P7P4FJRRJD4M52HHUH4SZRB2BQGUIVSLLWAOTA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 00:58:58.555237547 (537h29m58.426320582s) [noise=true , address=145.79.227.43:4159 , piece_id=UQBRTTTRHKZ7Y2P7P4FJRRJD4M52HHUH4SZRB2BQGUIVSLLWAOTA, seg_pos={0 0}, upload=471395]: Long tail cancellation due to optimal threshold Dec 28 00:58:58.555265921 (537h29m58.426348956s) [noise=true , address=145.79.227.44:4194 , piece_id=SI52P53FVLIGM5SFJXYZ2FF4MY6B3B72TDPAZMDHX4ITMPOINJBA, seg_pos={0 0}, upload=471395]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 28 00:58:58.555271050 (537h29m58.426354085s) [noise=true , address=145.79.227.44:4194 , piece_id=SI52P53FVLIGM5SFJXYZ2FF4MY6B3B72TDPAZMDHX4ITMPOINJBA, seg_pos={0 0}, upload=471395]: Long tail cancellation due to optimal threshold Dec 28 00:58:58.555285649 (537h29m58.426368684s) [seg_pos={0 0}, upload=471395]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:58:58.555354009 (537h29m58.426437044s) [upload=471395]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:58:58.555358517 (537h29m58.426441562s) [upload=471395]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:58:58.843284555 (537h29m58.71436759s) [upload=471395]: Done uploading segments... Dec 28 00:58:58.843307820 (537h29m58.714390855s) [upload=471395]: Upload finished. err: <nil> Duration: 30.002991963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 28.594730237s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:59:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 28.594745296s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 37.303481459s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:58:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 37.303518159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X....] duration: 7.634159965s trace: https://jaeger.dev.storj.io/trace/43b8ae0fc43abb1d FAIL (2025-12-28 00:58:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:54:39.571036892 (537h25m39.442119937s) [upload=471336]: Starting upload Dec 28 00:54:39.571112146 (537h25m39.442195181s) [upload=471336]: Done starting upload Dec 28 00:54:39.571146471 (537h25m39.442229516s) [upload=471336]: Uploading segments... Dec 28 00:54:39.571352564 (537h25m39.442435609s) [upload=471336]: Got next segment. Inline: false Dec 28 00:54:39.571389333 (537h25m39.442472379s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:54:39.571394974 (537h25m39.442478019s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:54:39.819105876 (537h25m39.690188922s) [seg_pos={0 0}, upload=471336]: Begin upload segment... Dec 28 00:54:39.819160561 (537h25m39.690243596s) [seg_pos={0 0}, upload=471336]: Done begin upload segment. Dec 28 00:54:39.819212550 (537h25m39.690295585s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819219323 (537h25m39.690302358s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819792885 (537h25m39.69087592s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820079420 (537h25m39.691162455s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820343273 (537h25m39.691426308s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820600693 (537h25m39.691683728s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820888231 (537h25m39.691971266s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821290857 (537h25m39.692373892s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821671392 (537h25m39.692754427s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823121984 (537h25m39.69420502s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823437135 (537h25m39.69452018s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.824628483 (537h25m39.695711528s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826070598 (537h25m39.697153644s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826550502 (537h25m39.697633548s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.828033787 (537h25m39.699116822s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.829612443 (537h25m39.700695478s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:41.535470148 (537h25m41.406553183s) [upload=471336]: Next returned nil segment Dec 28 00:54:41.535483433 (537h25m41.406566468s) [upload=471336]: Waiting for error group managing segments... Dec 28 00:54:41.825793310 (537h25m41.696876355s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.839695201 (537h25m41.710778246s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.840397630 (537h25m41.711480695s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.842129898 (537h25m41.713212933s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.848645064 (537h25m41.719728099s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.854569996 (537h25m41.725653041s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.865492594 (537h25m41.736575629s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.010495543 (537h25m41.881578588s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.040443720 (537h25m41.911526765s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.045037658 (537h25m41.916120703s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.048156847 (537h25m41.919239882s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.117720708 (537h25m41.988803753s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119361613 (537h25m41.990444658s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119374126 (537h25m41.990457161s) [seg_pos={0 0}, upload=471336]: Segment reached optimal threshold of 13 pieces. Dec 28 00:54:42.119578986 (537h25m41.990662021s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119588113 (537h25m41.990671158s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119598444 (537h25m41.990681479s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119608703 (537h25m41.990691748s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119752948 (537h25m41.990835993s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119761714 (537h25m41.990844749s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119784458 (537h25m41.990867503s) [seg_pos={0 0}, upload=471336]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:54:42.119881543 (537h25m41.990964588s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:54:42.119919064 (537h25m41.991002109s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:54:42.419014862 (537h25m42.290097907s) [upload=471336]: Done uploading segments... Dec 28 00:54:42.419031904 (537h25m42.290114939s) [upload=471336]: Upload finished. err: <nil> Duration: 7.634180263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X...] duration: 8.542422794s trace: https://jaeger.dev.storj.io/trace/43b8ae0fc43abb1d FAIL (2025-12-28 00:57:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:54:39.571036892 (537h25m39.442119937s) [upload=471336]: Starting upload Dec 28 00:54:39.571112146 (537h25m39.442195181s) [upload=471336]: Done starting upload Dec 28 00:54:39.571146471 (537h25m39.442229516s) [upload=471336]: Uploading segments... Dec 28 00:54:39.571352564 (537h25m39.442435609s) [upload=471336]: Got next segment. Inline: false Dec 28 00:54:39.571389333 (537h25m39.442472379s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:54:39.571394974 (537h25m39.442478019s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:54:39.819105876 (537h25m39.690188922s) [seg_pos={0 0}, upload=471336]: Begin upload segment... Dec 28 00:54:39.819160561 (537h25m39.690243596s) [seg_pos={0 0}, upload=471336]: Done begin upload segment. Dec 28 00:54:39.819212550 (537h25m39.690295585s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819219323 (537h25m39.690302358s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819792885 (537h25m39.69087592s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820079420 (537h25m39.691162455s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820343273 (537h25m39.691426308s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820600693 (537h25m39.691683728s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820888231 (537h25m39.691971266s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821290857 (537h25m39.692373892s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821671392 (537h25m39.692754427s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823121984 (537h25m39.69420502s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823437135 (537h25m39.69452018s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.824628483 (537h25m39.695711528s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826070598 (537h25m39.697153644s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826550502 (537h25m39.697633548s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.828033787 (537h25m39.699116822s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.829612443 (537h25m39.700695478s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:41.535470148 (537h25m41.406553183s) [upload=471336]: Next returned nil segment Dec 28 00:54:41.535483433 (537h25m41.406566468s) [upload=471336]: Waiting for error group managing segments... Dec 28 00:54:41.825793310 (537h25m41.696876355s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.839695201 (537h25m41.710778246s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.840397630 (537h25m41.711480695s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.842129898 (537h25m41.713212933s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.848645064 (537h25m41.719728099s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.854569996 (537h25m41.725653041s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.865492594 (537h25m41.736575629s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.010495543 (537h25m41.881578588s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.040443720 (537h25m41.911526765s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.045037658 (537h25m41.916120703s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.048156847 (537h25m41.919239882s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.117720708 (537h25m41.988803753s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119361613 (537h25m41.990444658s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119374126 (537h25m41.990457161s) [seg_pos={0 0}, upload=471336]: Segment reached optimal threshold of 13 pieces. Dec 28 00:54:42.119578986 (537h25m41.990662021s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119588113 (537h25m41.990671158s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119598444 (537h25m41.990681479s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119608703 (537h25m41.990691748s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119752948 (537h25m41.990835993s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119761714 (537h25m41.990844749s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119784458 (537h25m41.990867503s) [seg_pos={0 0}, upload=471336]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:54:42.119881543 (537h25m41.990964588s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:54:42.119919064 (537h25m41.991002109s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:54:42.419014862 (537h25m42.290097907s) [upload=471336]: Done uploading segments... Dec 28 00:54:42.419031904 (537h25m42.290114939s) [upload=471336]: Upload finished. err: <nil> Duration: 8.542460538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 25.428896479s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:57:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 25.428925915s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X..] duration: 5.967756741s trace: https://jaeger.dev.storj.io/trace/43b8ae0fc43abb1d FAIL (2025-12-28 00:56:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:54:39.571036892 (537h25m39.442119937s) [upload=471336]: Starting upload Dec 28 00:54:39.571112146 (537h25m39.442195181s) [upload=471336]: Done starting upload Dec 28 00:54:39.571146471 (537h25m39.442229516s) [upload=471336]: Uploading segments... Dec 28 00:54:39.571352564 (537h25m39.442435609s) [upload=471336]: Got next segment. Inline: false Dec 28 00:54:39.571389333 (537h25m39.442472379s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:54:39.571394974 (537h25m39.442478019s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:54:39.819105876 (537h25m39.690188922s) [seg_pos={0 0}, upload=471336]: Begin upload segment... Dec 28 00:54:39.819160561 (537h25m39.690243596s) [seg_pos={0 0}, upload=471336]: Done begin upload segment. Dec 28 00:54:39.819212550 (537h25m39.690295585s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819219323 (537h25m39.690302358s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819792885 (537h25m39.69087592s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820079420 (537h25m39.691162455s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820343273 (537h25m39.691426308s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820600693 (537h25m39.691683728s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820888231 (537h25m39.691971266s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821290857 (537h25m39.692373892s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821671392 (537h25m39.692754427s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823121984 (537h25m39.69420502s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823437135 (537h25m39.69452018s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.824628483 (537h25m39.695711528s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826070598 (537h25m39.697153644s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826550502 (537h25m39.697633548s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.828033787 (537h25m39.699116822s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.829612443 (537h25m39.700695478s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:41.535470148 (537h25m41.406553183s) [upload=471336]: Next returned nil segment Dec 28 00:54:41.535483433 (537h25m41.406566468s) [upload=471336]: Waiting for error group managing segments... Dec 28 00:54:41.825793310 (537h25m41.696876355s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.839695201 (537h25m41.710778246s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.840397630 (537h25m41.711480695s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.842129898 (537h25m41.713212933s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.848645064 (537h25m41.719728099s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.854569996 (537h25m41.725653041s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.865492594 (537h25m41.736575629s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.010495543 (537h25m41.881578588s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.040443720 (537h25m41.911526765s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.045037658 (537h25m41.916120703s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.048156847 (537h25m41.919239882s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.117720708 (537h25m41.988803753s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119361613 (537h25m41.990444658s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119374126 (537h25m41.990457161s) [seg_pos={0 0}, upload=471336]: Segment reached optimal threshold of 13 pieces. Dec 28 00:54:42.119578986 (537h25m41.990662021s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119588113 (537h25m41.990671158s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119598444 (537h25m41.990681479s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119608703 (537h25m41.990691748s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119752948 (537h25m41.990835993s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119761714 (537h25m41.990844749s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119784458 (537h25m41.990867503s) [seg_pos={0 0}, upload=471336]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:54:42.119881543 (537h25m41.990964588s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:54:42.119919064 (537h25m41.991002109s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:54:42.419014862 (537h25m42.290097907s) [upload=471336]: Done uploading segments... Dec 28 00:54:42.419031904 (537h25m42.290114939s) [upload=471336]: Upload finished. err: <nil> Duration: 5.967777971s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 24.984547463s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:56:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 24.984574986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 31.414472817s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:56:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 31.414500752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X.] duration: 13.054402846s trace: https://jaeger.dev.storj.io/trace/43b8ae0fc43abb1d FAIL (2025-12-28 00:55:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:54:39.571036892 (537h25m39.442119937s) [upload=471336]: Starting upload Dec 28 00:54:39.571112146 (537h25m39.442195181s) [upload=471336]: Done starting upload Dec 28 00:54:39.571146471 (537h25m39.442229516s) [upload=471336]: Uploading segments... Dec 28 00:54:39.571352564 (537h25m39.442435609s) [upload=471336]: Got next segment. Inline: false Dec 28 00:54:39.571389333 (537h25m39.442472379s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:54:39.571394974 (537h25m39.442478019s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:54:39.819105876 (537h25m39.690188922s) [seg_pos={0 0}, upload=471336]: Begin upload segment... Dec 28 00:54:39.819160561 (537h25m39.690243596s) [seg_pos={0 0}, upload=471336]: Done begin upload segment. Dec 28 00:54:39.819212550 (537h25m39.690295585s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819219323 (537h25m39.690302358s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819792885 (537h25m39.69087592s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820079420 (537h25m39.691162455s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820343273 (537h25m39.691426308s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820600693 (537h25m39.691683728s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820888231 (537h25m39.691971266s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821290857 (537h25m39.692373892s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821671392 (537h25m39.692754427s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823121984 (537h25m39.69420502s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823437135 (537h25m39.69452018s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.824628483 (537h25m39.695711528s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826070598 (537h25m39.697153644s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826550502 (537h25m39.697633548s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.828033787 (537h25m39.699116822s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.829612443 (537h25m39.700695478s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:41.535470148 (537h25m41.406553183s) [upload=471336]: Next returned nil segment Dec 28 00:54:41.535483433 (537h25m41.406566468s) [upload=471336]: Waiting for error group managing segments... Dec 28 00:54:41.825793310 (537h25m41.696876355s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.839695201 (537h25m41.710778246s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.840397630 (537h25m41.711480695s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.842129898 (537h25m41.713212933s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.848645064 (537h25m41.719728099s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.854569996 (537h25m41.725653041s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.865492594 (537h25m41.736575629s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.010495543 (537h25m41.881578588s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.040443720 (537h25m41.911526765s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.045037658 (537h25m41.916120703s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.048156847 (537h25m41.919239882s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.117720708 (537h25m41.988803753s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119361613 (537h25m41.990444658s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119374126 (537h25m41.990457161s) [seg_pos={0 0}, upload=471336]: Segment reached optimal threshold of 13 pieces. Dec 28 00:54:42.119578986 (537h25m41.990662021s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119588113 (537h25m41.990671158s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119598444 (537h25m41.990681479s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119608703 (537h25m41.990691748s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119752948 (537h25m41.990835993s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119761714 (537h25m41.990844749s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119784458 (537h25m41.990867503s) [seg_pos={0 0}, upload=471336]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:54:42.119881543 (537h25m41.990964588s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:54:42.119919064 (537h25m41.991002109s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:54:42.419014862 (537h25m42.290097907s) [upload=471336]: Done uploading segments... Dec 28 00:54:42.419031904 (537h25m42.290114939s) [upload=471336]: Upload finished. err: <nil> Duration: 13.054440638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 32.339611964s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:55:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 32.339628304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X] duration: 30.003653216s trace: https://jaeger.dev.storj.io/trace/43b8ae0fc43abb1d FAIL (2025-12-28 00:55:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:54:39.571036892 (537h25m39.442119937s) [upload=471336]: Starting upload Dec 28 00:54:39.571112146 (537h25m39.442195181s) [upload=471336]: Done starting upload Dec 28 00:54:39.571146471 (537h25m39.442229516s) [upload=471336]: Uploading segments... Dec 28 00:54:39.571352564 (537h25m39.442435609s) [upload=471336]: Got next segment. Inline: false Dec 28 00:54:39.571389333 (537h25m39.442472379s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:54:39.571394974 (537h25m39.442478019s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:54:39.819105876 (537h25m39.690188922s) [seg_pos={0 0}, upload=471336]: Begin upload segment... Dec 28 00:54:39.819160561 (537h25m39.690243596s) [seg_pos={0 0}, upload=471336]: Done begin upload segment. Dec 28 00:54:39.819212550 (537h25m39.690295585s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819219323 (537h25m39.690302358s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819792885 (537h25m39.69087592s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820079420 (537h25m39.691162455s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820343273 (537h25m39.691426308s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820600693 (537h25m39.691683728s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820888231 (537h25m39.691971266s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821290857 (537h25m39.692373892s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821671392 (537h25m39.692754427s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823121984 (537h25m39.69420502s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823437135 (537h25m39.69452018s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.824628483 (537h25m39.695711528s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826070598 (537h25m39.697153644s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826550502 (537h25m39.697633548s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.828033787 (537h25m39.699116822s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.829612443 (537h25m39.700695478s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:41.535470148 (537h25m41.406553183s) [upload=471336]: Next returned nil segment Dec 28 00:54:41.535483433 (537h25m41.406566468s) [upload=471336]: Waiting for error group managing segments... Dec 28 00:54:41.825793310 (537h25m41.696876355s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.839695201 (537h25m41.710778246s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.840397630 (537h25m41.711480695s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.842129898 (537h25m41.713212933s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.848645064 (537h25m41.719728099s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.854569996 (537h25m41.725653041s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.865492594 (537h25m41.736575629s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.010495543 (537h25m41.881578588s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.040443720 (537h25m41.911526765s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.045037658 (537h25m41.916120703s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.048156847 (537h25m41.919239882s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.117720708 (537h25m41.988803753s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119361613 (537h25m41.990444658s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119374126 (537h25m41.990457161s) [seg_pos={0 0}, upload=471336]: Segment reached optimal threshold of 13 pieces. Dec 28 00:54:42.119578986 (537h25m41.990662021s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119588113 (537h25m41.990671158s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119598444 (537h25m41.990681479s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119608703 (537h25m41.990691748s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119752948 (537h25m41.990835993s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119761714 (537h25m41.990844749s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119784458 (537h25m41.990867503s) [seg_pos={0 0}, upload=471336]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:54:42.119881543 (537h25m41.990964588s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:54:42.119919064 (537h25m41.991002109s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:54:42.419014862 (537h25m42.290097907s) [upload=471336]: Done uploading segments... Dec 28 00:54:42.419031904 (537h25m42.290114939s) [upload=471336]: Upload finished. err: <nil> Duration: 30.003680808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 22.526216332s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:54:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 22.526245176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 36.818949848s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:54:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 36.818979235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.] duration: 6.252786764s trace: https://jaeger.dev.storj.io/trace/a15d2188dd84399 FAIL (2025-12-28 00:54:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ) context deadline exceeded ======LOGS====== 8 00:53:17.596126537 (537h24m17.467209582s) [upload=471314]: Starting upload Dec 28 00:53:17.596180970 (537h24m17.467264015s) [upload=471314]: Done starting upload Dec 28 00:53:17.596280680 (537h24m17.467363715s) [upload=471314]: Uploading segments... Dec 28 00:53:17.596460573 (537h24m17.467543618s) [upload=471314]: Got next segment. Inline: false Dec 28 00:53:17.596530095 (537h24m17.46761314s) [upload=471314]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:53:17.596537921 (537h24m17.467620966s) [upload=471314]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:53:17.839703236 (537h24m17.710786281s) [seg_pos={0 0}, upload=471314]: Begin upload segment... Dec 28 00:53:17.839754034 (537h24m17.710837069s) [seg_pos={0 0}, upload=471314]: Done begin upload segment. Dec 28 00:53:17.839771627 (537h24m17.710854662s) [noise=true , address=145.79.227.36:4518 , piece_id=DMCMIDRIAHSWNFYFJHVWQENWA7BBJLYP4PR3INHPHMP6X7UUQ53Q, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.839780484 (537h24m17.710863519s) [noise=true , address=145.79.227.24:4583 , piece_id=ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.840133737 (537h24m17.711216782s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.840760450 (537h24m17.711843485s) [noise=true , address=145.79.227.4:4186 , piece_id=VUV3VHGS3RXOGUXCUF6FERGV7V6SGZVHOYUHJVDTWW4FR2V6EG2A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841064990 (537h24m17.712148025s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841364650 (537h24m17.712447695s) [noise=true , address=45.91.116.8:4047 , piece_id=HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841624686 (537h24m17.712707731s) [noise=true , address=145.79.227.39:4015 , piece_id=JCJKYVDAZTDP63JDC4R5X7Y5OMMCR56QEJAPCNAFVCNGGIMZU37A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841918625 (537h24m17.71300166s) [noise=true , address=77.90.31.16:4735 , piece_id=FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.842222734 (537h24m17.713305769s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.843130152 (537h24m17.714213187s) [noise=true , address=145.79.227.11:4731 , piece_id=QBEUKPCJ2R2TSHFBQXVQSPJQQJXROHBP55FAEN5WQYEV5GPU2E2Q, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.843293894 (537h24m17.714376929s) [noise=true , address=45.91.116.4:4939 , piece_id=HUCVIO45HEIJFGGLZANB23TMHKDRCGEHE3V7Q34C54IH353YSHOA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.844176515 (537h24m17.71525955s) [noise=true , address=45.91.116.13:4153 , piece_id=7K7R5QY63QDZSFWLRK7ZKVGAK3RPABQTRVBQSI4G5BFPOV7HMOJQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.845030610 (537h24m17.716113656s) [noise=true , address=145.79.227.3:4150 , piece_id=5DN3ECBYRXSTFGG4RCPUF24LDKTCHBUTUHJAELSHOVBDR5MOMG4A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.846051455 (537h24m17.71713449s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.847369986 (537h24m17.718453021s) [noise=true , address=145.79.227.19:4079 , piece_id=I3UO3747YY7EBKVDFISXM3N5DPA42BEBFXMHSQXAY4ZP2M2UXCMA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.849424648 (537h24m17.720507683s) [noise=true , address=45.91.116.7:4010 , piece_id=JHSUSJMDKVHUSII4D34MYSLRNZFDF5BLBKWIFC6LX3KKZRCSWKLQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:19.453212734 (537h24m19.324295769s) [upload=471314]: Next returned nil segment Dec 28 00:53:19.453225879 (537h24m19.324308914s) [upload=471314]: Waiting for error group managing segments... Dec 28 00:53:19.836508783 (537h24m19.707591818s) [noise=true , address=145.79.227.19:4079 , piece_id=I3UO3747YY7EBKVDFISXM3N5DPA42BEBFXMHSQXAY4ZP2M2UXCMA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.839183255 (537h24m19.71026629s) [noise=true , address=145.79.227.24:4583 , piece_id=ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.840850039 (537h24m19.711933084s) [noise=true , address=145.79.227.11:4731 , piece_id=QBEUKPCJ2R2TSHFBQXVQSPJQQJXROHBP55FAEN5WQYEV5GPU2E2Q, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.848938079 (537h24m19.720021115s) [noise=true , address=145.79.227.4:4186 , piece_id=VUV3VHGS3RXOGUXCUF6FERGV7V6SGZVHOYUHJVDTWW4FR2V6EG2A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.851779490 (537h24m19.722862525s) [noise=true , address=145.79.227.3:4150 , piece_id=5DN3ECBYRXSTFGG4RCPUF24LDKTCHBUTUHJAELSHOVBDR5MOMG4A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.861869403 (537h24m19.732952439s) [noise=true , address=145.79.227.36:4518 , piece_id=DMCMIDRIAHSWNFYFJHVWQENWA7BBJLYP4PR3INHPHMP6X7UUQ53Q, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.887942351 (537h24m19.759025386s) [noise=true , address=145.79.227.39:4015 , piece_id=JCJKYVDAZTDP63JDC4R5X7Y5OMMCR56QEJAPCNAFVCNGGIMZU37A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.022520831 (537h24m19.893603866s) [noise=true , address=77.90.31.16:4735 , piece_id=FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.088879838 (537h24m19.959962873s) [noise=true , address=45.91.116.4:4939 , piece_id=HUCVIO45HEIJFGGLZANB23TMHKDRCGEHE3V7Q34C54IH353YSHOA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.109535119 (537h24m19.980618164s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.150146541 (537h24m20.021229586s) [noise=true , address=45.91.116.8:4047 , piece_id=HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.172140200 (537h24m20.043223245s) [noise=true , address=45.91.116.13:4153 , piece_id=7K7R5QY63QDZSFWLRK7ZKVGAK3RPABQTRVBQSI4G5BFPOV7HMOJQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.177725925 (537h24m20.04880896s) [noise=true , address=45.91.116.7:4010 , piece_id=JHSUSJMDKVHUSII4D34MYSLRNZFDF5BLBKWIFC6LX3KKZRCSWKLQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.177829362 (537h24m20.048912397s) [seg_pos={0 0}, upload=471314]: Segment reached optimal threshold of 13 pieces. Dec 28 00:53:20.178008814 (537h24m20.049091849s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection Dec 28 00:53:20.178015678 (537h24m20.049098733s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178120217 (537h24m20.049203252s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 00:53:20.178137339 (537h24m20.049220374s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178174620 (537h24m20.049257655s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 00:53:20.178185089 (537h24m20.049268124s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178201261 (537h24m20.049284306s) [seg_pos={0 0}, upload=471314]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:53:20.178293467 (537h24m20.049376502s) [upload=471314]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:53:20.178299308 (537h24m20.049382343s) [upload=471314]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:53:20.466730187 (537h24m20.337813252s) [upload=471314]: Done uploading segments... Dec 28 00:53:20.466756076 (537h24m20.337839111s) [upload=471314]: Upload finished. err: <nil> Duration: 6.252811281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [..X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX] duration: 30.002659318s trace: https://jaeger.dev.storj.io/trace/a15d2188dd84399 FAIL (2025-12-28 00:53:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ) context deadline exceeded ======LOGS====== 8 00:53:17.596126537 (537h24m17.467209582s) [upload=471314]: Starting upload Dec 28 00:53:17.596180970 (537h24m17.467264015s) [upload=471314]: Done starting upload Dec 28 00:53:17.596280680 (537h24m17.467363715s) [upload=471314]: Uploading segments... Dec 28 00:53:17.596460573 (537h24m17.467543618s) [upload=471314]: Got next segment. Inline: false Dec 28 00:53:17.596530095 (537h24m17.46761314s) [upload=471314]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:53:17.596537921 (537h24m17.467620966s) [upload=471314]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:53:17.839703236 (537h24m17.710786281s) [seg_pos={0 0}, upload=471314]: Begin upload segment... Dec 28 00:53:17.839754034 (537h24m17.710837069s) [seg_pos={0 0}, upload=471314]: Done begin upload segment. Dec 28 00:53:17.839771627 (537h24m17.710854662s) [noise=true , address=145.79.227.36:4518 , piece_id=DMCMIDRIAHSWNFYFJHVWQENWA7BBJLYP4PR3INHPHMP6X7UUQ53Q, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.839780484 (537h24m17.710863519s) [noise=true , address=145.79.227.24:4583 , piece_id=ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.840133737 (537h24m17.711216782s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.840760450 (537h24m17.711843485s) [noise=true , address=145.79.227.4:4186 , piece_id=VUV3VHGS3RXOGUXCUF6FERGV7V6SGZVHOYUHJVDTWW4FR2V6EG2A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841064990 (537h24m17.712148025s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841364650 (537h24m17.712447695s) [noise=true , address=45.91.116.8:4047 , piece_id=HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841624686 (537h24m17.712707731s) [noise=true , address=145.79.227.39:4015 , piece_id=JCJKYVDAZTDP63JDC4R5X7Y5OMMCR56QEJAPCNAFVCNGGIMZU37A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841918625 (537h24m17.71300166s) [noise=true , address=77.90.31.16:4735 , piece_id=FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.842222734 (537h24m17.713305769s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.843130152 (537h24m17.714213187s) [noise=true , address=145.79.227.11:4731 , piece_id=QBEUKPCJ2R2TSHFBQXVQSPJQQJXROHBP55FAEN5WQYEV5GPU2E2Q, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.843293894 (537h24m17.714376929s) [noise=true , address=45.91.116.4:4939 , piece_id=HUCVIO45HEIJFGGLZANB23TMHKDRCGEHE3V7Q34C54IH353YSHOA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.844176515 (537h24m17.71525955s) [noise=true , address=45.91.116.13:4153 , piece_id=7K7R5QY63QDZSFWLRK7ZKVGAK3RPABQTRVBQSI4G5BFPOV7HMOJQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.845030610 (537h24m17.716113656s) [noise=true , address=145.79.227.3:4150 , piece_id=5DN3ECBYRXSTFGG4RCPUF24LDKTCHBUTUHJAELSHOVBDR5MOMG4A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.846051455 (537h24m17.71713449s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.847369986 (537h24m17.718453021s) [noise=true , address=145.79.227.19:4079 , piece_id=I3UO3747YY7EBKVDFISXM3N5DPA42BEBFXMHSQXAY4ZP2M2UXCMA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.849424648 (537h24m17.720507683s) [noise=true , address=45.91.116.7:4010 , piece_id=JHSUSJMDKVHUSII4D34MYSLRNZFDF5BLBKWIFC6LX3KKZRCSWKLQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:19.453212734 (537h24m19.324295769s) [upload=471314]: Next returned nil segment Dec 28 00:53:19.453225879 (537h24m19.324308914s) [upload=471314]: Waiting for error group managing segments... Dec 28 00:53:19.836508783 (537h24m19.707591818s) [noise=true , address=145.79.227.19:4079 , piece_id=I3UO3747YY7EBKVDFISXM3N5DPA42BEBFXMHSQXAY4ZP2M2UXCMA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.839183255 (537h24m19.71026629s) [noise=true , address=145.79.227.24:4583 , piece_id=ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.840850039 (537h24m19.711933084s) [noise=true , address=145.79.227.11:4731 , piece_id=QBEUKPCJ2R2TSHFBQXVQSPJQQJXROHBP55FAEN5WQYEV5GPU2E2Q, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.848938079 (537h24m19.720021115s) [noise=true , address=145.79.227.4:4186 , piece_id=VUV3VHGS3RXOGUXCUF6FERGV7V6SGZVHOYUHJVDTWW4FR2V6EG2A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.851779490 (537h24m19.722862525s) [noise=true , address=145.79.227.3:4150 , piece_id=5DN3ECBYRXSTFGG4RCPUF24LDKTCHBUTUHJAELSHOVBDR5MOMG4A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.861869403 (537h24m19.732952439s) [noise=true , address=145.79.227.36:4518 , piece_id=DMCMIDRIAHSWNFYFJHVWQENWA7BBJLYP4PR3INHPHMP6X7UUQ53Q, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.887942351 (537h24m19.759025386s) [noise=true , address=145.79.227.39:4015 , piece_id=JCJKYVDAZTDP63JDC4R5X7Y5OMMCR56QEJAPCNAFVCNGGIMZU37A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.022520831 (537h24m19.893603866s) [noise=true , address=77.90.31.16:4735 , piece_id=FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.088879838 (537h24m19.959962873s) [noise=true , address=45.91.116.4:4939 , piece_id=HUCVIO45HEIJFGGLZANB23TMHKDRCGEHE3V7Q34C54IH353YSHOA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.109535119 (537h24m19.980618164s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.150146541 (537h24m20.021229586s) [noise=true , address=45.91.116.8:4047 , piece_id=HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.172140200 (537h24m20.043223245s) [noise=true , address=45.91.116.13:4153 , piece_id=7K7R5QY63QDZSFWLRK7ZKVGAK3RPABQTRVBQSI4G5BFPOV7HMOJQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.177725925 (537h24m20.04880896s) [noise=true , address=45.91.116.7:4010 , piece_id=JHSUSJMDKVHUSII4D34MYSLRNZFDF5BLBKWIFC6LX3KKZRCSWKLQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.177829362 (537h24m20.048912397s) [seg_pos={0 0}, upload=471314]: Segment reached optimal threshold of 13 pieces. Dec 28 00:53:20.178008814 (537h24m20.049091849s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection Dec 28 00:53:20.178015678 (537h24m20.049098733s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178120217 (537h24m20.049203252s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 00:53:20.178137339 (537h24m20.049220374s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178174620 (537h24m20.049257655s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 00:53:20.178185089 (537h24m20.049268124s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178201261 (537h24m20.049284306s) [seg_pos={0 0}, upload=471314]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:53:20.178293467 (537h24m20.049376502s) [upload=471314]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:53:20.178299308 (537h24m20.049382343s) [upload=471314]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:53:20.466730187 (537h24m20.337813252s) [upload=471314]: Done uploading segments... Dec 28 00:53:20.466756076 (537h24m20.337839111s) [upload=471314]: Upload finished. err: <nil> Duration: 30.002670039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XX] duration: 30.003316271s trace: https://jaeger.dev.storj.io/trace/708eb75639035a8c FAIL (2025-12-28 00:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:52:47.359336763 (537h23m47.230419798s) [upload=471306]: Starting upload Dec 28 00:52:47.359401146 (537h23m47.230484181s) [upload=471306]: Done starting upload Dec 28 00:52:47.359491108 (537h23m47.230574143s) [upload=471306]: Uploading segments... Dec 28 00:52:47.359581650 (537h23m47.230664685s) [upload=471306]: Got next segment. Inline: false Dec 28 00:52:47.359608050 (537h23m47.230691085s) [upload=471306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:52:47.359613971 (537h23m47.230697006s) [upload=471306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:52:47.600479008 (537h23m47.471562043s) [seg_pos={0 0}, upload=471306]: Begin upload segment... Dec 28 00:52:47.600532019 (537h23m47.471615064s) [seg_pos={0 0}, upload=471306]: Done begin upload segment. Dec 28 00:52:47.600563359 (537h23m47.471646404s) [noise=true , address=145.79.227.47:4267 , piece_id=YXH5RFKI3GNLZ3YIYKGGQVYVEL2GWOAVU3T2W6WOKMLITC3KKMUQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.600593847 (537h23m47.471676882s) [noise=true , address=77.90.31.21:4878 , piece_id=PTNDJHQAIP37KRSOPXEL55RYSET5CTKK4FPOEVUIDFYEKNL5DK2Q, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.601081305 (537h23m47.47216434s) [noise=true , address=145.79.227.42:4123 , piece_id=BOW7P7FSCFRIZZUQ6Z2DHRNV7FQPB5KUHIFFPA6Z2G7CPAEVXFAQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.601474413 (537h23m47.472557459s) [noise=true , address=45.91.116.12:4225 , piece_id=PEUJCVV2DODGEGXN6WFVSOFXH3UYHQPSKRT5OIGK5ZN26EPIUZKQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.601937836 (537h23m47.473020871s) [noise=true , address=145.79.227.18:4007 , piece_id=O7DNTMFXJIJPBQAZQNY3L7GAJYYVBZ4LTFVP5E5JMUHHYIP2SCTQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.602328059 (537h23m47.473476097s) [noise=true , address=145.79.227.5:4223 , piece_id=7PANZFYGH2TMSH7HMNUXZXXD775PHFV4LXZ3XG2BNICMBTRJ3DNA, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.602793585 (537h23m47.473876631s) [noise=true , address=45.91.116.6:4973 , piece_id=63C2VPR37BWBYA2DFM7LGQ4UPEQCNDHPMJ6RHOJ2S4U4AKH4K4YQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.603194969 (537h23m47.474278004s) [noise=true , address=145.79.227.52:4446 , piece_id=G4CYPJKTJJCV6P7KP5B3PWWLILQDNUMAR6XERJT5QX4BD7AG3NIQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.603626421 (537h23m47.474709456s) [noise=true , address=45.91.116.13:4156 , piece_id=RZQLRWJYVUUN74KILOAIBZCAA5LJNPU44ZELG5CYHN73GHGVBG2A, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.605542981 (537h23m47.476626026s) [noise=true , address=145.79.227.37:4567 , piece_id=JNN2IMAUY3UILQLOMX5PLMY7XU4I4CJCZDLB5ZDAFPWOZI2B4PSA, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.607444282 (537h23m47.478527317s) [noise=true , address=45.91.116.11:4117 , piece_id=M6IB6PGYXCFNCF26R3HYSBAJYMRPA6IAZU73WAGOH2ON2Q5AJEVQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.609215523 (537h23m47.480298569s) [noise=true , address=145.79.227.17:4970 , piece_id=SEIVKYWAZVVNANA4LQAESMRUGIR4LXR3EFMNKNMDZY7Y2KPIQHIA, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.610837733 (537h23m47.481920778s) [noise=true , address=45.91.116.10:4084 , piece_id=HEHX7TL2ZBOBXDPUFRMVZXSEDJPMWN2ESJVHOGB42WICX6NOZ33Q, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.612350693 (537h23m47.483433738s) [noise=true , address=145.79.227.15:4875 , piece_id=7GBN3KRYHOQXYRBFRY7U4CFRCQO7Y6A4ILJOBNQDPFXSZEEKK27Q, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.612963571 (537h23m47.484046616s) [noise=true , address=45.91.116.14:4190 , piece_id=M7RMUNKLAYGIBCG72QKJMEYDDEW6TCQRBK2P3IU5YQLWZGCAHOHQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.614289896 (537h23m47.485372931s) [noise=true , address=145.79.227.31:4798 , piece_id=AOSC4E6A6FSEKVWDMX7QH4MHJR7DGIDQBOYYF2R4PFZ5B5GFKDPQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:49.312692584 (537h23m49.183775619s) [upload=471306]: Next returned nil segment Dec 28 00:52:49.312710978 (537h23m49.183794013s) [upload=471306]: Waiting for error group managing segments... Dec 28 00:52:49.593196308 (537h23m49.464279363s) [noise=true , address=145.79.227.52:4446 , piece_id=G4CYPJKTJJCV6P7KP5B3PWWLILQDNUMAR6XERJT5QX4BD7AG3NIQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.607920475 (537h23m49.479003511s) [noise=true , address=145.79.227.5:4223 , piece_id=7PANZFYGH2TMSH7HMNUXZXXD775PHFV4LXZ3XG2BNICMBTRJ3DNA, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.608503756 (537h23m49.479586791s) [noise=true , address=145.79.227.18:4007 , piece_id=O7DNTMFXJIJPBQAZQNY3L7GAJYYVBZ4LTFVP5E5JMUHHYIP2SCTQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.611487658 (537h23m49.482570693s) [noise=true , address=145.79.227.37:4567 , piece_id=JNN2IMAUY3UILQLOMX5PLMY7XU4I4CJCZDLB5ZDAFPWOZI2B4PSA, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.615678298 (537h23m49.486761343s) [noise=true , address=145.79.227.31:4798 , piece_id=AOSC4E6A6FSEKVWDMX7QH4MHJR7DGIDQBOYYF2R4PFZ5B5GFKDPQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.617878427 (537h23m49.488961462s) [noise=true , address=145.79.227.17:4970 , piece_id=SEIVKYWAZVVNANA4LQAESMRUGIR4LXR3EFMNKNMDZY7Y2KPIQHIA, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.625630799 (537h23m49.496713834s) [noise=true , address=145.79.227.42:4123 , piece_id=BOW7P7FSCFRIZZUQ6Z2DHRNV7FQPB5KUHIFFPA6Z2G7CPAEVXFAQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.632602184 (537h23m49.503685219s) [noise=true , address=145.79.227.15:4875 , piece_id=7GBN3KRYHOQXYRBFRY7U4CFRCQO7Y6A4ILJOBNQDPFXSZEEKK27Q, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.645619371 (537h23m49.516702406s) [noise=true , address=145.79.227.47:4267 , piece_id=YXH5RFKI3GNLZ3YIYKGGQVYVEL2GWOAVU3T2W6WOKMLITC3KKMUQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.821062483 (537h23m49.692145518s) [noise=true , address=45.91.116.14:4190 , piece_id=M7RMUNKLAYGIBCG72QKJMEYDDEW6TCQRBK2P3IU5YQLWZGCAHOHQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.822478129 (537h23m49.693561164s) [noise=true , address=45.91.116.6:4973 , piece_id=63C2VPR37BWBYA2DFM7LGQ4UPEQCNDHPMJ6RHOJ2S4U4AKH4K4YQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.889949516 (537h23m49.761032561s) [noise=true , address=45.91.116.13:4156 , piece_id=RZQLRWJYVUUN74KILOAIBZCAA5LJNPU44ZELG5CYHN73GHGVBG2A, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.960834711 (537h23m49.831917746s) [noise=true , address=45.91.116.11:4117 , piece_id=M6IB6PGYXCFNCF26R3HYSBAJYMRPA6IAZU73WAGOH2ON2Q5AJEVQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.960854940 (537h23m49.831937975s) [seg_pos={0 0}, upload=471306]: Segment reached optimal threshold of 13 pieces. Dec 28 00:52:49.961035984 (537h23m49.83211905s) [noise=true , address=45.91.116.12:4225 , piece_id=PEUJCVV2DODGEGXN6WFVSOFXH3UYHQPSKRT5OIGK5ZN26EPIUZKQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:51668->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:51668->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51668->45.91.116.12:4225: use of closed network connection Dec 28 00:52:49.961046635 (537h23m49.832129671s) [noise=true , address=45.91.116.12:4225 , piece_id=PEUJCVV2DODGEGXN6WFVSOFXH3UYHQPSKRT5OIGK5ZN26EPIUZKQ, seg_pos={0 0}, upload=471306]: Long tail cancellation due to optimal threshold Dec 28 00:52:49.961162756 (537h23m49.832245801s) [noise=true , address=45.91.116.10:4084 , piece_id=HEHX7TL2ZBOBXDPUFRMVZXSEDJPMWN2ESJVHOGB42WICX6NOZ33Q, seg_pos={0 0}, upload=471306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 00:52:49.961188536 (537h23m49.832271571s) [noise=true , address=45.91.116.10:4084 , piece_id=HEHX7TL2ZBOBXDPUFRMVZXSEDJPMWN2ESJVHOGB42WICX6NOZ33Q, seg_pos={0 0}, upload=471306]: Long tail cancellation due to optimal threshold Dec 28 00:52:49.961316829 (537h23m49.832399864s) [noise=true , address=77.90.31.21:4878 , piece_id=PTNDJHQAIP37KRSOPXEL55RYSET5CTKK4FPOEVUIDFYEKNL5DK2Q, seg_pos={0 0}, upload=471306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:52:49.961323091 (537h23m49.832406126s) [noise=true , address=77.90.31.21:4878 , piece_id=PTNDJHQAIP37KRSOPXEL55RYSET5CTKK4FPOEVUIDFYEKNL5DK2Q, seg_pos={0 0}, upload=471306]: Long tail cancellation due to optimal threshold Dec 28 00:52:49.961345704 (537h23m49.832428739s) [seg_pos={0 0}, upload=471306]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:52:49.961404577 (537h23m49.832487612s) [upload=471306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:52:49.961408945 (537h23m49.83249198s) [upload=471306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:52:50.252714419 (537h23m50.123797454s) [upload=471306]: Done uploading segments... Dec 28 00:52:50.252733645 (537h23m50.12381668s) [upload=471306]: Upload finished. err: <nil> Duration: 30.003326722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 26.923912933s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:52:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 26.923932209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.X] duration: 30.003346207s trace: https://jaeger.dev.storj.io/trace/47c5c3723baca7db FAIL (2025-12-28 00:52:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:52:17.135722231 (537h23m17.006805266s) [upload=471301]: Starting upload Dec 28 00:52:17.135803355 (537h23m17.00688639s) [upload=471301]: Done starting upload Dec 28 00:52:17.135879921 (537h23m17.006962976s) [upload=471301]: Uploading segments... Dec 28 00:52:17.136034265 (537h23m17.00711731s) [upload=471301]: Got next segment. Inline: false Dec 28 00:52:17.136068831 (537h23m17.007151866s) [upload=471301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:52:17.136074051 (537h23m17.007157086s) [upload=471301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:52:17.376948455 (537h23m17.24803149s) [seg_pos={0 0}, upload=471301]: Begin upload segment... Dec 28 00:52:17.377039659 (537h23m17.248122694s) [seg_pos={0 0}, upload=471301]: Done begin upload segment. Dec 28 00:52:17.377062713 (537h23m17.248145748s) [noise=true , address=145.79.227.9:4367 , piece_id=MWYOCHRIWT5TFT3NAGMXBRQXPEBNEFHNVRNKLCE6ZPWFRFOTWKQA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.377101116 (537h23m17.248184151s) [noise=true , address=145.79.227.45:4230 , piece_id=YR3TTTRONDH6J2LL3QC6KCON63DYH6C6AW7F74BD5UQWIQLULEUQ, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.377836747 (537h23m17.248919792s) [noise=true , address=145.79.227.34:4906 , piece_id=XHDUXXMVO7PJO6SZXGMAAKYPDKMBPZW3APLLWH4FHR4IA5RFUGCQ, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.378303385 (537h23m17.24938642s) [noise=true , address=77.90.31.15:4700 , piece_id=6EU7II5ZQPYHIJNXGX6CSBWPPNJ6KQGZKGDF3IDLYR7ELNVIZ2RQ, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.378906233 (537h23m17.249989268s) [noise=true , address=145.79.227.19:4077 , piece_id=XCB5JZA23RMKACPWJ25NPHSXQVUYZGSDN46XU4OJ5ROLWER45AIQ, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.379374665 (537h23m17.25045771s) [noise=true , address=145.79.227.37:4566 , piece_id=JDPV77NA47A7HT6NESM3NUACXZGVQ45U6ODFPH6IUSKAFRXXWREA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.379877432 (537h23m17.250960467s) [noise=true , address=45.91.116.8:4045 , piece_id=RYAV5W76SEZF6ZMJDJ34SK7NV3LZK6FJB2DLX4IH6ZLTWV6ZCFOA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.380321648 (537h23m17.251404683s) [noise=true , address=145.79.227.31:4798 , piece_id=YW6EI7AYQG54OTO3B5FPQOEVEYYCFOJNRNILYJKWX3HB5QM7AR3Q, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.380804648 (537h23m17.251887693s) [noise=true , address=45.91.116.14:4191 , piece_id=P7C3WUPLCNSCFAQAIXWW6J5OGRLG7MIEPHNHYJVWZ5B6GQXRFMDA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.381160696 (537h23m17.252243731s) [noise=true , address=45.91.116.6:4973 , piece_id=GKRQJZK4BN22TQZ5N4B7L7YUIB5LE2UQ5HNCNNYGRFAZMDOLNWFA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.381471317 (537h23m17.252554352s) [noise=true , address=145.79.227.12:4765 , piece_id=R7P4O3J2VVLONXTA7GWOYDPG6HOPYF2Z5XUWFHZSBET4JJEKS7DA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.382733150 (537h23m17.253816195s) [noise=true , address=145.79.227.16:4911 , piece_id=OSOUQ35KVDTYRSZNI5TDHTHWMJMGNDZLNONAKVXCFXC4URBKWB2Q, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.383014376 (537h23m17.254097421s) [noise=true , address=145.79.227.47:4266 , piece_id=IN2L2F4GYUARBLUKUYVTMGRR2KA3IXB5RLH6OCPFYLUXGFQ2UR2A, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.384138717 (537h23m17.255221752s) [noise=true , address=145.79.227.52:4445 , piece_id=PXGUP24MMQNOQOOG2CNPBCBVLFOQD6PI4MX6S7VJDWCQNDDRFHJQ, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.385323422 (537h23m17.256406468s) [noise=true , address=145.79.227.36:4518 , piece_id=MWCXCRSAY6B3Z3ZENFP6RHIUR3IVZGJ5ACDJA5ONDPKH4INKMWTA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.386702920 (537h23m17.257785955s) [noise=true , address=45.91.116.12:4228 , piece_id=JO3QXHYWHTQ6UO4YXGBGQ5WKUMWVC3FX2SIDC4IZ76F4WOXEPNQA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:19.099667165 (537h23m18.97075021s) [upload=471301]: Next returned nil segment Dec 28 00:52:19.099691070 (537h23m18.970774115s) [upload=471301]: Waiting for error group managing segments... Dec 28 00:52:19.386244696 (537h23m19.257327741s) [noise=true , address=145.79.227.12:4765 , piece_id=R7P4O3J2VVLONXTA7GWOYDPG6HOPYF2Z5XUWFHZSBET4JJEKS7DA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.387712541 (537h23m19.258795576s) [noise=true , address=145.79.227.36:4518 , piece_id=MWCXCRSAY6B3Z3ZENFP6RHIUR3IVZGJ5ACDJA5ONDPKH4INKMWTA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.388003735 (537h23m19.25908677s) [noise=true , address=145.79.227.45:4230 , piece_id=YR3TTTRONDH6J2LL3QC6KCON63DYH6C6AW7F74BD5UQWIQLULEUQ, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.393358752 (537h23m19.264441807s) [noise=true , address=145.79.227.31:4798 , piece_id=YW6EI7AYQG54OTO3B5FPQOEVEYYCFOJNRNILYJKWX3HB5QM7AR3Q, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.405272007 (537h23m19.276355052s) [noise=true , address=145.79.227.9:4367 , piece_id=MWYOCHRIWT5TFT3NAGMXBRQXPEBNEFHNVRNKLCE6ZPWFRFOTWKQA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.405342932 (537h23m19.276425977s) [noise=true , address=145.79.227.52:4445 , piece_id=PXGUP24MMQNOQOOG2CNPBCBVLFOQD6PI4MX6S7VJDWCQNDDRFHJQ, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.412559583 (537h23m19.283642618s) [noise=true , address=145.79.227.37:4566 , piece_id=JDPV77NA47A7HT6NESM3NUACXZGVQ45U6ODFPH6IUSKAFRXXWREA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.421288234 (537h23m19.292371269s) [noise=true , address=145.79.227.19:4077 , piece_id=XCB5JZA23RMKACPWJ25NPHSXQVUYZGSDN46XU4OJ5ROLWER45AIQ, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.421672977 (537h23m19.292756012s) [noise=true , address=145.79.227.16:4911 , piece_id=OSOUQ35KVDTYRSZNI5TDHTHWMJMGNDZLNONAKVXCFXC4URBKWB2Q, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.426927062 (537h23m19.298010097s) [noise=true , address=145.79.227.47:4266 , piece_id=IN2L2F4GYUARBLUKUYVTMGRR2KA3IXB5RLH6OCPFYLUXGFQ2UR2A, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.620948981 (537h23m19.492032016s) [noise=true , address=45.91.116.14:4191 , piece_id=P7C3WUPLCNSCFAQAIXWW6J5OGRLG7MIEPHNHYJVWZ5B6GQXRFMDA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.621400401 (537h23m19.492483436s) [noise=true , address=77.90.31.15:4700 , piece_id=6EU7II5ZQPYHIJNXGX6CSBWPPNJ6KQGZKGDF3IDLYR7ELNVIZ2RQ, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.622225602 (537h23m19.493308648s) [noise=true , address=45.91.116.6:4973 , piece_id=GKRQJZK4BN22TQZ5N4B7L7YUIB5LE2UQ5HNCNNYGRFAZMDOLNWFA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.622233708 (537h23m19.493316743s) [seg_pos={0 0}, upload=471301]: Segment reached optimal threshold of 13 pieces. Dec 28 00:52:19.622474717 (537h23m19.493557752s) [noise=true , address=45.91.116.12:4228 , piece_id=JO3QXHYWHTQ6UO4YXGBGQ5WKUMWVC3FX2SIDC4IZ76F4WOXEPNQA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 00:52:19.622481660 (537h23m19.493564705s) [noise=true , address=45.91.116.12:4228 , piece_id=JO3QXHYWHTQ6UO4YXGBGQ5WKUMWVC3FX2SIDC4IZ76F4WOXEPNQA, seg_pos={0 0}, upload=471301]: Long tail cancellation due to optimal threshold Dec 28 00:52:19.622624161 (537h23m19.493707206s) [noise=true , address=145.79.227.34:4906 , piece_id=XHDUXXMVO7PJO6SZXGMAAKYPDKMBPZW3APLLWH4FHR4IA5RFUGCQ, seg_pos={0 0}, upload=471301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 00:52:19.622644370 (537h23m19.493727405s) [noise=true , address=145.79.227.34:4906 , piece_id=XHDUXXMVO7PJO6SZXGMAAKYPDKMBPZW3APLLWH4FHR4IA5RFUGCQ, seg_pos={0 0}, upload=471301]: Long tail cancellation due to optimal threshold Dec 28 00:52:19.622694214 (537h23m19.493777249s) [noise=true , address=45.91.116.8:4045 , piece_id=RYAV5W76SEZF6ZMJDJ34SK7NV3LZK6FJB2DLX4IH6ZLTWV6ZCFOA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 00:52:19.622702169 (537h23m19.493785204s) [noise=true , address=45.91.116.8:4045 , piece_id=RYAV5W76SEZF6ZMJDJ34SK7NV3LZK6FJB2DLX4IH6ZLTWV6ZCFOA, seg_pos={0 0}, upload=471301]: Long tail cancellation due to optimal threshold Dec 28 00:52:19.622742918 (537h23m19.493825953s) [seg_pos={0 0}, upload=471301]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:52:19.622876130 (537h23m19.493959176s) [upload=471301]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:52:19.622881582 (537h23m19.493964627s) [upload=471301]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:52:19.907721984 (537h23m19.77880502s) [upload=471301]: Done uploading segments... Dec 28 00:52:19.907745829 (537h23m19.778828884s) [upload=471301]: Upload finished. err: <nil> Duration: 30.003367727s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)