Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 8.006094497s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:29:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 8.006125485s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................XX...................X......] duration: 6.743949174s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:29:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.743987898s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 324.348721ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6a3cce270cf7783d FAIL (2025-11-11 17:29:09 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42728->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 324.372266ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.41109134s trace: https://jaeger.dev.storj.io/trace/37aef011ca2a9db9 FAIL (2025-11-11 17:28:47 UTC) orbiter warning: context deadline exceeded Duration: 3.411141264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.889739841s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:28:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.889769467s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X................................XX...................X.....] duration: 6.626853987s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:28:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.626886389s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 305.744303ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6a3cce270cf7783d FAIL (2025-11-11 17:28:05 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42728->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 305.775154ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.434489533s trace: https://jaeger.dev.storj.io/trace/37aef011ca2a9db9 FAIL (2025-11-11 17:27:56 UTC) orbiter warning: context deadline exceeded Duration: 3.43451992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X................................XX...................X....] duration: 6.864707655s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:27:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.864742101s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.024991969s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:27:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.025010965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X................................XX...................X...] duration: 6.996529452s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:27:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.996563797s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 312.544654ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6a3cce270cf7783d FAIL (2025-11-11 17:27:02 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42728->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 312.579229ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.677676605s trace: https://jaeger.dev.storj.io/trace/37aef011ca2a9db9 FAIL (2025-11-11 17:26:48 UTC) orbiter warning: context deadline exceeded Duration: 3.677703648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.092322966s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:26:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.092349857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X................................XX...................X..] duration: 7.029727368s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:26:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.029748559s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 232.22593ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/6a3cce270cf7783d FAIL (2025-11-11 17:26:06 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:42728->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 232.255867ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 5.000736902s trace: https://jaeger.dev.storj.io/trace/37aef011ca2a9db9 FAIL (2025-11-11 17:25:56 UTC) orbiter warning: context deadline exceeded Duration: 5.000884745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X................................XX...................X.] duration: 7.872590514s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:25:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.872628196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.065983988s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:25:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.066014307s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001855243s trace: https://jaeger.dev.storj.io/trace/37aef011ca2a9db9 FAIL (2025-11-11 17:25:35 UTC) orbiter warning: context deadline exceeded Duration: 5.001878989s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................................X] duration: 1m0.261197247s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:25:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.261221943s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X................................XX...................X] duration: 30.268309833s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:25:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.268339559s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X................................XX...................] duration: 7.339590175s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:23:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 7.33962445s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.938370374s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:23:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.938402086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X......X................................XX..................] duration: 8.47253593s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:23:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 8.472555247s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.86332944s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:22:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.863372342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.334277175s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:22:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.334301012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X......X................................XX.................] duration: 9.0411649s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:21:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 9.041201309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.155042026s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:21:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.15514975s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X......X................................XX................] duration: 7.077442369s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:20:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 7.077475873s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.018544623s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:20:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.018592835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.054804s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:20:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.054894251s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X......X................................XX...............] duration: 7.069982921s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:20:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 7.070028759s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X......X................................XX..............] duration: 6.849094207s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:19:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 6.849162259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.49622543s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:19:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.496350969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X......X................................XX.............] duration: 7.571309727s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:19:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 7.571434546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X......X................................XX............] duration: 7.089629633s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:18:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 7.089663358s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X......X................................XX...........] duration: 6.479213412s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:17:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 6.479243298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.553143332s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:17:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.553170553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.827665202s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:17:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.827694288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X......X................................XX..........] duration: 8.807818204s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:16:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 8.807862919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.835417599s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:16:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.835492341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.355517794s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:16:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.355538421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X......X................................XX.........] duration: 7.522899096s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:16:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 7.523090381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X......X................................XX........] duration: 6.766451372s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 6.76648241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.362749382s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:15:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.362780973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X......X................................XX.......] duration: 7.037298853s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:15:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 7.037329781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.213163689s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:14:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.213192866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X......X................................XX......] duration: 8.048576322s trace: https://jaeger.dev.storj.io/trace/32f48b4f1799253 FAIL (2025-11-11 17:14:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, Piece ID: EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ) context deadline exceeded; piecestore: (Node ID: 12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, Piece ID: BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ) context deadline exceeded; piecestore: (Node ID: 12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, Piece ID: XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q) context deadline exceeded; piecestore: (Node ID: 12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, Piece ID: BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA) context deadline exceeded ======LOGS====== 11 17:09:17.808254746 (57h5m16.359725207s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:09:17.808259335 (57h5m16.359729786s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:09:18.065943994 (57h5m16.617414455s) [seg_pos={0 0}, upload=50295]: Begin upload segment... Nov 11 17:09:18.066007305 (57h5m16.617477766s) [seg_pos={0 0}, upload=50295]: Done begin upload segment. Nov 11 17:09:18.066048393 (57h5m16.617518854s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067020474 (57h5m16.618490925s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.067697985 (57h5m16.619168436s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.069137075 (57h5m16.620607536s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.070196713 (57h5m16.621667164s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.074296229 (57h5m16.62576668s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.076492341 (57h5m16.627962802s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.078065547 (57h5m16.629536008s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.080540098 (57h5m16.632010549s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.081056011 (57h5m16.632526472s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.084048509 (57h5m16.63551897s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.085709442 (57h5m16.637179893s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.086524133 (57h5m16.637994594s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.087118326 (57h5m16.638588777s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.088483044 (57h5m16.639953505s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:18.089429967 (57h5m16.640900428s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Uploading piece... Nov 11 17:09:19.843494253 (57h5m18.394964714s) [upload=50295]: Next returned nil segment Nov 11 17:09:19.843513309 (57h5m18.39498379s) [upload=50295]: Waiting for error group managing segments... Nov 11 17:09:20.178022832 (57h5m18.729493333s) [noise=true , address=45.91.116.4:4940 , piece_id=7HY6FK6RA4AERNEGFIIUNGRKG5UI6QJ6D6NATTFZKKWDSKTGY3PQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.191212378 (57h5m18.742682849s) [noise=true , address=77.90.31.12:4592 , piece_id=VRSDWRPJKAWWU3X2DYQSQB5QLTYPHFH6RMIRNSRKQIK75ZF2UAHA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.214652943 (57h5m18.766123394s) [noise=true , address=45.91.116.11:4118 , piece_id=GLBZIWAUC6OBA3EXQSFDVPQZ3B7AOMA6PSBUJU7GU6CLHRW45NMA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.228934699 (57h5m18.78040515s) [noise=true , address=45.91.116.7:4009 , piece_id=B2ONDGKYMCRX3DSMNSKX34BRJLRXRJD35BKL4FE7FMUFKLGRGERQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.257611417 (57h5m18.809081878s) [noise=true , address=45.91.116.6:4973 , piece_id=DGNIVMHVC476IJ4HVBKEGQQY2ADVWBF5TQ272H3EUBXDXMFRR7OQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.286180058 (57h5m18.837650519s) [noise=true , address=45.91.116.13:4153 , piece_id=NK22OZKAZRF746YPBHO27E6QGR2VJI33DAF5TW7FSUMDK7VIU5UQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.361426805 (57h5m18.912897266s) [noise=true , address=45.91.116.5:4297 , piece_id=PF7UO5J6C5KHI34EIL4ITP37MSVVNGWW3YC63TJQPWSDNCGIEXYA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:20.375330360 (57h5m18.926800821s) [noise=true , address=77.90.31.15:4697 , piece_id=BCMQWJVGOJ7CDB3WCCTICLSUI7YCO5EONPOB6AHD47M3NSDTW27Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:31.518915065 (57h5m30.070385516s) [noise=true , address=145.79.227.22:4509 , piece_id=DADDKGVC2EMAEIFHT5GQSQA2BZYWLSW2HFUFDCJWLKTPV5IRPDYQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:34.758887248 (57h5m33.310357699s) [noise=true , address=145.79.227.9:4365 , piece_id=BRYKBIIQ6QWQNGGU56TBFY4HLYWFLEMB5OVMEXQJLNQ3WZY4IYJQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.165420831 (57h5m33.716891282s) [noise=true , address=145.79.227.6:4258 , piece_id=BTDKFHFFSLOMRC72LKIL323XKNZOVAYRJEUDMC7TR6XOBQS7F2DA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:35.922385673 (57h5m34.473856124s) [noise=true , address=145.79.227.17:4970 , piece_id=EOCEGELS4TE6RFJHB6AOGCCW3WUIL4BW2BAQFKSC3FHRW6MYHBHQ, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.840964696 (57h5m43.392435147s) [noise=true , address=145.79.227.14:4837 , piece_id=XFMTSEG7WYTWOWD6YEFQ6FBVNCZE57TCWTJQGPU3UUMKBZMGJY3Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: <nil> Nov 11 17:09:44.841004482 (57h5m43.392474943s) [seg_pos={0 0}, upload=50295]: Segment reached optimal threshold of 13 pieces. Nov 11 17:09:44.841717200 (57h5m43.393187671s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU): protocol: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36884->145.79.227.8:4331: use of closed network connection Nov 11 17:09:44.841729202 (57h5m43.393199683s) [noise=true , address=145.79.227.8:4331 , piece_id=WXFEUINZGYNRURKROFWHSPXW4EVNJYJC2SJTUV27GD3C5ADIE22A, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842051686 (57h5m43.393522147s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46518->145.79.227.27:4690: use of closed network connection Nov 11 17:09:44.842059662 (57h5m43.393530123s) [noise=true , address=145.79.227.27:4690 , piece_id=6MGPMMRUHIHDZRSDIELYKDRYYMAGDAFUNSUWUFGFGCWRM4RFQS5Q, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842244383 (57h5m43.393714844s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4): protocol: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40918->145.79.227.13:4802: use of closed network connection Nov 11 17:09:44.842252308 (57h5m43.393722759s) [noise=true , address=145.79.227.13:4802 , piece_id=LR6HIAJBJHU453NXMSGXW75EUNUC3SXWFH3UFJES6XVW76JK22SA, seg_pos={0 0}, upload=50295]: Long tail cancellation due to optimal threshold Nov 11 17:09:44.842269892 (57h5m43.393740353s) [seg_pos={0 0}, upload=50295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 17:09:44.842338914 (57h5m43.393809375s) [upload=50295]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 17:09:44.842343502 (57h5m43.393813963s) [upload=50295]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 17:09:45.141069414 (57h5m43.692539875s) [upload=50295]: Done uploading segments... Nov 11 17:09:45.141088591 (57h5m43.692559042s) [upload=50295]: Upload finished. err: <nil> Duration: 8.048602752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 7.406110565s trace: https://jaeger.dev.storj.io/trace/78d654ae55ebb605 FAIL (2025-11-11 17:13:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== , upload=49863]: Uploading piece... Nov 11 16:38:52.691951338 (56h34m51.243421799s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.692798161 (56h34m51.244268622s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.693354811 (56h34m51.244825272s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.694778972 (56h34m51.246249424s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.695733631 (56h34m51.247204082s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.697236272 (56h34m51.248706723s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.699562031 (56h34m51.251032512s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.700319393 (56h34m51.251789854s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.703427281 (56h34m51.254897742s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.706289372 (56h34m51.257759833s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.710676576 (56h34m51.262147047s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.714572143 (56h34m51.266042605s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.717196280 (56h34m51.268666731s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.718265035 (56h34m51.269735496s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721110724 (56h34m51.272581185s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.721545772 (56h34m51.273016233s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:52.724518373 (56h34m51.275988834s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Uploading piece... Nov 11 16:38:54.085017947 (56h34m52.636488398s) [upload=49863]: Next returned nil segment Nov 11 16:38:54.085039368 (56h34m52.63650983s) [upload=49863]: Waiting for error group managing segments... Nov 11 16:38:54.448618452 (56h34m53.000088913s) [noise=true , address=64.49.13.27:4957 , piece_id=TQLAVLU3IGZK3YIDGNAE76UV4N2CZSKGBOXXUBUEKB7VU6U7GKIA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.510488886 (56h34m53.061959347s) [noise=true , address=5.231.73.2:4011 , piece_id=A3ETOZBKHUEZM7H2HUCJEXWBQWCENTB2JJ2Y4CAYAJFA56H223GA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.540378382 (56h34m53.091848843s) [noise=true , address=5.231.73.3:4006 , piece_id=XTC5CC5VCTWLDEK7ECYKYXH7OXB3W2YDKSQLUFP2TSPZPXVFVJMA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.639349766 (56h34m53.190820227s) [noise=true , address=77.90.31.12:4593 , piece_id=YEDKJ4VY5CH6S4ZVYFC25DPQ44JUECUZVKOSR32F2UBABM44RCJQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.685463415 (56h34m53.236933876s) [noise=true , address=45.91.116.10:4093 , piece_id=MLAWZTYAVGYIE3GN46W7UQPBOSRZO5B3TDB5LG7PBO5VEJJUEISA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.689681578 (56h34m53.241152029s) [noise=true , address=77.90.31.12:4596 , piece_id=7WBOOXOEMO6B6PCSRSVNB7TJGCTBYMRERQ3KYMSPGK3GG7L24V2Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.703150185 (56h34m53.254620646s) [noise=true , address=77.90.31.14:4685 , piece_id=Y6PPDBNWNBFTAVB3J2G55TO74QQCLZABEG2WCWBAGQHNPAK5A65Q, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.733346886 (56h34m53.284817337s) [noise=true , address=77.90.31.21:4891 , piece_id=DBZ7Q5PSWDNH5AZS46TR2PR3FTEKCPUQNKUCM7NFWCJMGNTVSLWA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.767761068 (56h34m53.319231539s) [noise=true , address=77.90.31.17:4798 , piece_id=WK5VGN7GQ4SDKFPS6UDXMRI4GHUP6446J3LSPXRQEXXZSJIAZEQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:54.775188923 (56h34m53.326659374s) [noise=true , address=45.91.116.11:4127 , piece_id=2SZU3UQCX5DMK6HQ65RSTDJRGLA4PX3F5RPDCMC3BMT253JMWKPA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:38:55.020594547 (56h34m53.572065009s) [noise=true , address=77.90.31.11:4588 , piece_id=55J7TS7ENKPGKEX2PDDGXGFYIBTQQTZOXR3PS53EMKRT75Z4OYQQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:44.897670757 (56h35m43.449141218s) [noise=true , address=145.79.227.13:4833 , piece_id=XYX3PWBC5UPGMKGQTWFOKSKF2A3TPRWF6WL3BFQCV47MCHHPRWDQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: <nil> Nov 11 16:39:51.611845606 (56h35m50.163316067s) [noise=true , address=145.79.227.24:4605 , piece_id=WDCDSD76JWYZFR2T6VHL7VURC4XEGU55S46Y2FRW2IFL4MPV32HQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:145.79.227.24:4605): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.611966777 (56h35m50.163437228s) [noise=true , address=145.79.227.5:4239 , piece_id=M2BAQCFL2MUJLGFYLBQGO6ILXUIEYPNUAO6PV2KJXFBQ3ZWKALFA, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38478->145.79.227.5:4239: use of closed network connection Nov 11 16:39:51.611976646 (56h35m50.163447107s) [noise=true , address=145.79.227.16:4942 , piece_id=DBHLVMAFDMIDNRDKOS2PAXRFDYSC2PAJ6R2WG7WYUL3OSIQ2GYIQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:145.79.227.16:4942): protocol: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50396->145.79.227.16:4942: use of closed network connection Nov 11 16:39:51.612113146 (56h35m50.163583607s) [noise=true , address=145.79.227.43:4163 , piece_id=U4XAQYS5MYPYS36R24X4WIY5RQOILUM2KSOP6MBRVMKR6SBWZVCQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:145.79.227.43:4163): protocol: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58248->145.79.227.43:4163: use of closed network connection Nov 11 16:39:51.612425420 (56h35m50.163895891s) [noise=true , address=145.79.227.39:4025 , piece_id=QJ5ETZX3OGBVDE3Z475KW3CGSTKGBTYXGTYVIIZZZMJASFTCOVSQ, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12bUYkG2a33thoYtLYL1KAJfJh2mDsbMTnPdK2bfEJARwMz5Lv2, address:145.79.227.39:4025): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 16:39:51.612485836 (56h35m50.163956297s) [noise=true , address=145.79.227.49:4360 , piece_id=2JG45UW7ZTAGCDJTFYSBT5L6PCS5NHFOLKR3IOCVCG6JZLXUB42A, seg_pos={0 0}, upload=49863]: Done uploading piece. err: ecclient: upload failed (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk, address:145.79.227.49:4360): protocol: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55084->145.79.227.49:4360: use of closed network connection Nov 11 16:39:51.612524740 (56h35m50.163995201s) [seg_pos={0 0}, upload=49863]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 16:39:51.900972958 (56h35m50.452443419s) [upload=49863]: Done uploading segments... Nov 11 16:39:51.901028663 (56h35m50.452499114s) [upload=49863]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 7.40613348s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)