Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.890966285s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:48:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.890999258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......XX...................X...............................] duration: 7.75017337s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:47:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.750211143s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 71.495804ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:47:46 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 71.523827ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 47.065373ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:47:24 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 47.094819ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X.........X.....................] duration: 3.982559251s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:47:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.98259043s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.048161519s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:47:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.048192147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........XX...................X..............................] duration: 6.691165051s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:46:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.691195039s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 50.558595ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:46:43 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 50.58235ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.887513279s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:46:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.887590687s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X.........X....................] duration: 3.968920078s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:46:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.969095763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.919711736s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:45:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.919731803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........XX...................X.............................] duration: 6.00053543s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:45:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.000564658s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 52.543264ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:45:36 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 52.571677ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 54.56303ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:45:13 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 54.58826ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................X.........X...................] duration: 3.982852741s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:45:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.98288381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........XX...................X............................] duration: 6.628577352s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:44:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.62861269s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................X.........X..................] duration: 3.929124831s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:44:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.929147153s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.079390758s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:44:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.079419682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........XX...................X...........................] duration: 7.023485333s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:44:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.02353117s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................X.........X.................] duration: 3.965400876s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:44:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.965432597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.022440272s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:44:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.022470349s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............XX...................X..........................] duration: 6.590270725s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:44:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.5903046s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 222.162747ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/610c86de212f0acb FAIL (2025-11-11 17:44:01 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:58138->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 222.192123ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................X.........X................] duration: 4.343903378s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:43:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.343930831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.843196996s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:43:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.843226081s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 296.229635ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/57d0a45d9b4d754d FAIL (2025-11-11 17:43:39 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55054->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 297.003278ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................X.........X...............] duration: 4.241012332s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:43:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.241034664s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 325.562012ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/57d0a45d9b4d754d FAIL (2025-11-11 17:43:07 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55054->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 325.599382ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.249804978s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:43:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.249838983s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............XX...................X.........................] duration: 6.767237817s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:42:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.767267593s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 319.003964ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/57d0a45d9b4d754d FAIL (2025-11-11 17:42:36 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55054->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 319.050411ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................X.........X..............] duration: 3.940098344s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:42:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.940129615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............XX...................X........................] duration: 6.290992143s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:42:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.29101546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 12.508550006s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:42:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 12.508584673s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................X.........X.............] duration: 3.991691723s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:41:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.991721299s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............XX...................X.......................] duration: 7.365346013s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:41:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.365371602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.150243559s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:41:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.150273426s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 305.756106ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/57d0a45d9b4d754d FAIL (2025-11-11 17:41:22 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55054->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 305.7851ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.442043449s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:41:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.44207499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................X.........X............] duration: 4.30760485s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:41:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.307627765s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................X.........X...........] duration: 3.939646185s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:40:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.939715987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.458714608s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:40:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.458738723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................XX...................X......................] duration: 6.247924209s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:40:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.247967271s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 315.909152ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/57d0a45d9b4d754d FAIL (2025-11-11 17:40:27 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:55054->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 315.927005ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................X.........X..........] duration: 3.907506684s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:40:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.907539576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.26556962s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:40:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.265624945s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................X.........X.........] duration: 3.95648016s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:39:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.956508112s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................XX...................X.....................] duration: 6.745493305s trace: https://jaeger.dev.storj.io/trace/2ae4efabccb1e953 FAIL (2025-11-11 17:39:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== =true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286486361 (57h20m35.837956853s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.286886683 (57h20m35.838357144s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287210671 (57h20m35.838681122s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.287624469 (57h20m35.83909493s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.288025963 (57h20m35.839496424s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.290439209 (57h20m35.84190968s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.291989691 (57h20m35.843460182s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.293416858 (57h20m35.844887329s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.294752232 (57h20m35.846222693s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.296046536 (57h20m35.847516997s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.297414251 (57h20m35.848884702s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.298646517 (57h20m35.850116978s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:37.299951132 (57h20m35.851421593s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Uploading piece... Nov 11 17:24:39.092245733 (57h20m37.643716194s) [upload=50533]: Next returned nil segment Nov 11 17:24:39.092287312 (57h20m37.643757763s) [upload=50533]: Waiting for error group managing segments... Nov 11 17:24:39.382395001 (57h20m37.933865462s) [noise=true , address=77.90.31.21:4878 , piece_id=W3F5SI6CWEQHN2P7NMX6UD4K6PT3V42OH55W6GOUWC4WX6LPIV3Q, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.455480663 (57h20m38.006951114s) [noise=true , address=45.91.116.5:4297 , piece_id=QWPLSS75TKE35BCJRRQJHSNK7JW7IUWESIO6NWHJ3BFVNDJHY45A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.464789719 (57h20m38.01626019s) [noise=true , address=77.90.31.10:4517 , piece_id=TS6FNQSTNS4DM52HGLVHLMQKUBOPM4NGFW26CWYSUVYZEN4IWQVQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.469662748 (57h20m38.021133209s) [noise=true , address=77.90.31.10:4519 , piece_id=LRO2AD77U7SIPLG4J7T6HHAYHWFP6C4CR22TDH3W424MZ7ZJAOEQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.498671969 (57h20m38.05014243s) [noise=true , address=45.91.116.8:4048 , piece_id=S44GFHAGY2H2OSCL2DXWVNWZOCSPZ246LSHU4ADINK6YOAEEGXQA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.501425191 (57h20m38.052895642s) [noise=true , address=77.90.31.11:4554 , piece_id=O3Q24O5NNLI3MB2JJAHDHYO3TB456VWOKT5MEHRCFE57ZFF4AWMQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.504427819 (57h20m38.05589828s) [noise=true , address=45.91.116.14:4190 , piece_id=2AKMNPPU7BCUJLIUW3YTURVTGEA36WYQ7ZFTYJAL25SHVM25LW5A, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:24:39.551590748 (57h20m38.103061209s) [noise=true , address=77.90.31.18:4805 , piece_id=SCJMTHHKXMUJMTDDFLAEYAONRYAEIXTYRPSUOYSOKD2UVMUOYTDQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: <nil> Nov 11 17:25:06.310688327 (57h21m4.862158789s) [noise=true , address=145.79.227.24:4582 , piece_id=E7Z3O3BF5POKD2HLN7OXVRPP7HDBODW7ZPZCAE2T5IAPSI3QNDSA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.24:4582: use of closed network connection Nov 11 17:25:06.310892937 (57h21m4.862363418s) [noise=true , address=145.79.227.4:4186 , piece_id=2CYXYEXNOXVBULD4BJSVY46GWF6ZHG2MV75GT3ALF4DQQ4RKVNLA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:145.79.227.4:4186): protocol: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34448->145.79.227.4:4186: use of closed network connection Nov 11 17:25:06.311062821 (57h21m4.862533282s) [noise=true , address=145.79.227.9:4366 , piece_id=MV3T4QXQMUA6EZEDW6EDPGCW6KXDCHIOGV5TUUHSCFR7BJ32TSMA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58020->145.79.227.9:4366: use of closed network connection Nov 11 17:25:06.311290033 (57h21m4.862760494s) [noise=true , address=145.79.227.15:4874 , piece_id=EHJC5R2TARTZD3MENLNGGPJRQQ7FF37MPPK6GK3BA5HRODHXCVXA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36332->145.79.227.15:4874: use of closed network connection Nov 11 17:25:06.311445520 (57h21m4.862915981s) [noise=true , address=145.79.227.11:4732 , piece_id=RJFJOYB5QNUVPSP2GGSZCZYXT2YLW3A7WNMWD6LWM63U5E23UCZQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:145.79.227.11:4732): protocol: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52888->145.79.227.11:4732: use of closed network connection Nov 11 17:25:06.311752634 (57h21m4.863223096s) [noise=true , address=145.79.227.15:4875 , piece_id=QF63TTO4D7Y43UWFOL7I3R666WES5RCLZIAKQZLSXCYV4VGP4OVA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54886->145.79.227.15:4875: use of closed network connection Nov 11 17:25:06.311912859 (57h21m4.86338332s) [noise=true , address=145.79.227.10:4403 , piece_id=5W2LKW5RU35GOP4TTZXBBSEBIATG7S4XJJ43AEQX23KULVV2BGDA, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54030->145.79.227.10:4403: use of closed network connection Nov 11 17:25:06.313081224 (57h21m4.864551695s) [noise=true , address=145.79.227.32:4835 , piece_id=PWCQ3VNFTV7WNUVRFT33RAWRSWIGSLATVPN6GKSNWTRDA33TX6WQ, seg_pos={0 0}, upload=50533]: Done uploading piece. err: ecclient: upload failed (node:16vVfeoqCkbMYCRjMMWaKBnCvQBBJPMgwQtWTq9HKQrD9SS17y, address:145.79.227.32:4835): protocol: write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection; write tcp 188.166.225.170:59066->145.79.227.32:4835: use of closed network connection Nov 11 17:25:06.321852808 (57h21m4.87332362s) [seg_pos={0 0}, upload=50533]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 17:25:06.577043016 (57h21m5.128513477s) [upload=50533]: Done uploading segments... Nov 11 17:25:06.577095756 (57h21m5.128566217s) [upload=50533]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.745528091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.06905104s trace: https://jaeger.dev.storj.io/trace/4f734dac703b215c FAIL (2025-11-11 17:39:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 3s) [upload=50531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 17:24:30.831227993 (57h20m29.382698444s) [upload=50531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 17:24:31.088253086 (57h20m29.639723567s) [seg_pos={0 0}, upload=50531]: Begin upload segment... Nov 11 17:24:31.088323530 (57h20m29.639793991s) [seg_pos={0 0}, upload=50531]: Done begin upload segment. Nov 11 17:24:31.088341564 (57h20m29.639812025s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088383874 (57h20m29.639854325s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088963047 (57h20m29.640433509s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.088974811 (57h20m29.640445262s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089409167 (57h20m29.640879628s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.089912436 (57h20m29.641382887s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090302118 (57h20m29.641772569s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.090690397 (57h20m29.642160858s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091093295 (57h20m29.642563756s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091499437 (57h20m29.642969888s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.091945958 (57h20m29.643416419s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.092404641 (57h20m29.643875102s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093275440 (57h20m29.644745901s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.093992765 (57h20m29.645463226s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.094492548 (57h20m29.645963009s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095035701 (57h20m29.646506162s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.095683926 (57h20m29.647154387s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:31.096503125 (57h20m29.647973587s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Uploading piece... Nov 11 17:24:32.442047219 (57h20m30.993517681s) [upload=50531]: Next returned nil segment Nov 11 17:24:32.442065835 (57h20m30.993536296s) [upload=50531]: Waiting for error group managing segments... Nov 11 17:24:32.796089334 (57h20m31.347559795s) [noise=true , address=64.49.13.138:4492 , piece_id=ADGUTWJGEQQEIZOWEMNTXZWDTSTKFDZDMHCCP7V3N2PM7COGHTPA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.819767743 (57h20m31.371238204s) [noise=true , address=64.49.13.155:4918 , piece_id=TBCILMDOP6YUWBGJCTX2HAWJXOHTRLO2MTJUUUMEUFMV3WQCNTXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.850771400 (57h20m31.402241852s) [noise=true , address=5.231.73.4:4011 , piece_id=PRJXUMFDDJJBISXRMMEB6NVUECW7GR7JAUXHMOZZIGJ4MXJZCZMQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:32.885468311 (57h20m31.436938762s) [noise=true , address=5.231.73.2:4008 , piece_id=YCYVPZB22UKMAWWAELIYLZCKXMT4PGTB6NWX5RIFVULMVVJKZMHQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.049531193 (57h20m31.601001654s) [noise=true , address=77.90.31.13:4625 , piece_id=EPFGONAY57CVE7VVVOVA2FEUOBREHGZJ4D5RNHGXQ746UWKKEGRA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.100236507 (57h20m31.651706968s) [noise=true , address=5.231.73.5:4011 , piece_id=6ZMCOURWMTOPLCAIZTTLFK5KKSUGK5VQ6GCRA3V4K22SNFAX53YQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.133005356 (57h20m31.684475817s) [noise=true , address=77.90.31.17:4789 , piece_id=JESTGAPW3HM6RQDNO77RSXVYAZ3GHYICFDGNWHR2WEJZSITRPY6Q, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.155744286 (57h20m31.707214757s) [noise=true , address=77.90.31.10:4533 , piece_id=3WTHZLJLXNJBWTD73B5IAU3CPB5NCTHQ6O4LU4UA2CTOH5JKV7NQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.172983924 (57h20m31.724454405s) [noise=true , address=77.90.31.21:4899 , piece_id=5LQNPPNMIBYNQIXRKFJ4ZIIB7F2M6ATL4DOBYHAOBJ5CSYBSMSIQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:24:33.188744195 (57h20m31.740214656s) [noise=true , address=45.91.116.6:4008 , piece_id=I7FNDUHBCU4CQWSL3DL2DUHYPQG4FXPP55V5TK2NLMQD7OWO2QQA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.398303853 (57h21m26.949774314s) [noise=true , address=145.79.227.19:4112 , piece_id=NNRKCZ3IYG3G4AJ7IG4M7GJBRCNZRMSGDSIIMYZM7LVNRYDSWVYA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:28.884365750 (57h21m27.435836202s) [noise=true , address=145.79.227.49:4363 , piece_id=XMPZGQAVKSRRHRC5NOGPJ2HIFVFY4FXUPYCRCL5QA2HXSUW5Y6WA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: <nil> Nov 11 17:25:30.103707405 (57h21m28.655177886s) [noise=true , address=145.79.227.12:4785 , piece_id=FS5KNU6C7656KNEAX5VNBDF54PFOY2FAEMVDDZY6XIOTZDBSVZIA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:145.79.227.12:4785): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.103890353 (57h21m28.655360814s) [noise=true , address=145.79.227.35:4504 , piece_id=MW7FJVXM3TMTFU6DA22ND63HBZ6LRNST5RQYJYT3SSQCVBVIHVDA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:145.79.227.35:4504): protocol: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59348->145.79.227.35:4504: use of closed network connection Nov 11 17:25:30.104036220 (57h21m28.655506691s) [noise=true , address=145.79.227.6:4292 , piece_id=SGG5N2DHFWXUX7CG6T7TPHRVHO5WBSUGWB37JF77VYNHIX2NVDXA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104174363 (57h21m28.655644824s) [noise=true , address=145.79.227.36:4518 , piece_id=36YBS3K5A53575PYCAHBAS5YD2ZDT5F4SJCZEG2P2LFL3SC4XXOQ, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:145.79.227.36:4518): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104298620 (57h21m28.655769081s) [noise=true , address=145.79.227.37:4577 , piece_id=OPCUSDA7BPJKMLD77K2XHWCSGRLI7U5OERTHGD3DIWSCODPCFOFA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:12BiDVmGW3SkqH6gMGoJj2cT2SbR3DbMVTPSWQgHcx4scb6cN5q, address:145.79.227.37:4577): protocol: expected piece hash; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; write tcp 188.166.225.170:54530->145.79.227.37:4577: use of closed network connection; EOF Nov 11 17:25:30.104545881 (57h21m28.656016342s) [noise=true , address=145.79.227.20:4456 , piece_id=XLC4LJ4VGGAPPASUNYXP5WVVANHVHAHVLMILNER25S2S42WBX2FA, seg_pos={0 0}, upload=50531]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:145.79.227.20:4456): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 17:25:30.104582441 (57h21m28.656052902s) [seg_pos={0 0}, upload=50531]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 11 17:25:30.362669054 (57h21m28.914139515s) [upload=50531]: Done uploading segments... Nov 11 17:25:30.362714881 (57h21m28.914185342s) [upload=50531]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 6.069079614s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................X.........X........] duration: 4.19655714s trace: https://jaeger.dev.storj.io/trace/70917b61a670b5cb FAIL (2025-11-11 17:39:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.196625418s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)