Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.839845641s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:25:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.839944889s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.861055009s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:24:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.861085328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.844010854s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:23:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.844037395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.893145667s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:22:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.893176947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.949939457s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:21:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.949971228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.047840671s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:20:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 6.047874506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.772230576s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:19:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.772266184s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.766416628s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:18:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.76644888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.744949532s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:17:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.744983074s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.634272055s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:16:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.634294307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.804248107s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:15:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.804280818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.766904997s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:15:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.766934693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.532472496s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:14:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.532562697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.765171946s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:13:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.765208195s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.785833078s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:13:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.785880629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.878408604s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:12:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.87843737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.814177783s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:11:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.81420789s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.754831214s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:10:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.754864428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.82596591s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:09:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.825999124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.675105873s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:08:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.675134389s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.787987951s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:08:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.788014001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.645518926s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:07:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.645548854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.724713523s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:06:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.724734091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.813744117s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:06:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.813776087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.790444739s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:05:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.790478363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.901157123s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:05:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.901193091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.869074309s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:04:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.869100449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.796025268s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:04:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.796083859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.853063459s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:03:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.853099377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.954554702s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:02:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.954657428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.899475581s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:02:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.899505197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.950702163s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:01:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.950738553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.901672683s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 18:00:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.901884336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.902232259s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:59:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.902252227s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.891298427s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:59:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.891336079s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.752601066s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:58:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.752632334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.63049143s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:57:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.630520996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.874261697s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:56:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.87429073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.827521844s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:55:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.827563502s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.207811109s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:55:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 6.207844142s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.810096417s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:54:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.810129041s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.848379742s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:53:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.848413097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.983862424s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:53:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.983881751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.723257909s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:52:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.723281865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.739986741s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:51:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.740017269s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.938997473s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:50:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.939030304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.8721989s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:49:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.87222494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.870557366s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:49:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.870590569s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.890132583s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:48:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.890166427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.693478553s trace: https://jaeger.dev.storj.io/trace/588f79fa694594cd FAIL (2025-11-14 17:48:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== =true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.932591096 (129h39m25.484061557s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933009723 (129h39m25.484480184s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.933494356 (129h39m25.484964807s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934013114 (129h39m25.485483575s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.934466317 (129h39m25.485936778s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935075406 (129h39m25.486545867s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.935713682 (129h39m25.487184143s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936281162 (129h39m25.487751614s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.936764753 (129h39m25.488235204s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.937367801 (129h39m25.488838262s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938008171 (129h39m25.489478632s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938495690 (129h39m25.489966161s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.938970403 (129h39m25.490440864s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.939634298 (129h39m25.491104759s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:26.940134230 (129h39m25.491604691s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Uploading piece... Nov 14 17:43:28.308719353 (129h39m26.860189814s) [upload=114132]: Next returned nil segment Nov 14 17:43:28.308727148 (129h39m26.86019761s) [upload=114132]: Waiting for error group managing segments... Nov 14 17:43:28.707535031 (129h39m27.259005502s) [noise=true , address=64.49.13.138:4482 , piece_id=F7RIQMVY7JJ4JYHP3ELLYWJXHDVDAMXV2X4WFI4XEPAFECCJOW5Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.729035453 (129h39m27.280505914s) [noise=true , address=5.231.73.3:4005 , piece_id=RKYQMBKDH4ED3USXSLNOZ3XNW2NMBJGU6MYW4MMECDP4ZDNHNEAQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.823870691 (129h39m27.375341152s) [noise=true , address=145.79.227.7:4313 , piece_id=WOFI5TQ2VPB4TSY3AQOCFBBFGDADUXWEH3ACEK2ZGXLXIYVVAVTQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.831822353 (129h39m27.383292824s) [noise=true , address=145.79.227.24:4613 , piece_id=NTEWHVXHYHXL625G2K7LF5UFXDREMHOAM4VPYGVPXVTPIL3T6YYA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.861922060 (129h39m27.413392521s) [noise=true , address=145.79.227.15:4879 , piece_id=M7YCHEC44YBSTAV2POUWUBR3CMOJQKMJDA5QQPWP4EX332BHWORQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.872633908 (129h39m27.424104359s) [noise=true , address=145.79.227.33:4873 , piece_id=HBM6FDAXGSLSYDRWKQFSSP5YFUQ7UEIVLA3Z3N3JCVJGPLAAAO6Q, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.880731728 (129h39m27.432202189s) [noise=true , address=145.79.227.47:4297 , piece_id=VJYDTX4RCWLN6K5YBLTM4V7V3RECBABRN6SSKVQCFLAQVNYWETHQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892293294 (129h39m27.443763745s) [noise=true , address=145.79.227.39:4046 , piece_id=BNNJVVCSB35WBJ6XOIRYBAKORJUP5PKYZC7QZ75Z5AAGTQD4AWGQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.892630936 (129h39m27.444101397s) [noise=true , address=77.90.31.19:4857 , piece_id=Y4VKY7MVKOB5MTGR5BUY2JQHMVKSCUMFJDCHIO5ZVOIHTJRM7NRQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.898592309 (129h39m27.45006276s) [noise=true , address=77.90.31.13:4659 , piece_id=MBBCRIZJQAYI7SE3BBGNYGTXRRXLEITR43HSR23N3OB7YZO5GGRA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.912258492 (129h39m27.463728953s) [noise=true , address=45.91.116.8:4076 , piece_id=YOAG3RQBYN7P657HUJFBSLTTNVLK4VL5NHIGGW537KQZX2GKUSMQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.913089614 (129h39m27.464560076s) [noise=true , address=45.91.116.5:4327 , piece_id=CIGG6QKFAF26AM7OYJVLMMUWE4A63UGDC23TUKOYEPCAAKYTOXUQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.917757704 (129h39m27.469228155s) [noise=true , address=77.90.31.14:4675 , piece_id=GGYODPFIHDBX2EWYKVJOGN54GLRKO6LT4JOF256XIOGU53JABO4A, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925888257 (129h39m27.477358718s) [noise=true , address=45.91.116.7:4023 , piece_id=IFCLEJ7EO2OHS4WNWB3QJCAWNYCEDNWF5VLSH2CMWSHCXHXZ2FQA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: <nil> Nov 14 17:43:28.925917503 (129h39m27.477387964s) [seg_pos={0 0}, upload=114132]: Segment reached optimal threshold of 14 pieces. Nov 14 17:43:28.926197866 (129h39m27.477668327s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926208055 (129h39m27.477678516s) [noise=true , address=77.90.31.17:4776 , piece_id=5R52BQP4UF4UYYXJ63LBYU4PKGES67EK5GPGJJMOCHXV2HF2D2TQ, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926346128 (129h39m27.477816599s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124WMsnYaj19ZVofFQz8wmiDBf4XKneRuaZbCxMHyGjca92gmUD): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926366347 (129h39m27.477836808s) [noise=true , address=45.91.116.12:4247 , piece_id=RG7CNE77OXUFBCPA363CGSJ2CI3LBFIXZV44YKPHZKF4EFRKJMJA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926642973 (129h39m27.478113424s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pJWhSLza8WaMuvvDwH4nae57UxQmfCdDvAkQ3v25MgPYHoZXx): protocol: expected piece hash; context canceled; EOF Nov 14 17:43:28.926669233 (129h39m27.478139694s) [noise=true , address=45.91.116.5:4302 , piece_id=SOKXZQJUZGJ7YW2UIOE5UDMK2X6O7SGALIES5KIKV7SMNJRI4ZKA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926750088 (129h39m27.478220549s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f1rdiK9aQoWLnj5pLCs2GN5daEiA1H9QLZfpyxUcFe38pCafr): protocol: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35880->145.79.227.26:4664: use of closed network connection Nov 14 17:43:28.926766629 (129h39m27.47823709s) [noise=true , address=145.79.227.26:4664 , piece_id=DSLECGEGVINORH2DEYVDP2FN5HNZA3DNC2GOLXVTAXHJGQPGBPAA, seg_pos={0 0}, upload=114132]: Long tail cancellation due to optimal threshold Nov 14 17:43:28.926795143 (129h39m27.478265605s) [seg_pos={0 0}, upload=114132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 14 17:43:28.926891808 (129h39m27.478362279s) [upload=114132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 14 17:43:28.926898560 (129h39m27.478369022s) [upload=114132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 14 17:43:29.233362448 (129h39m27.784832909s) [upload=114132]: Done uploading segments... Nov 14 17:43:29.233387626 (129h39m27.784858077s) [upload=114132]: Upload finished. err: <nil> Duration: 5.693509173s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)