Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 3.514239903s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:31:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.514258559s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..X] duration: 30.002742206s trace: https://jaeger.dev.storj.io/trace/291c428b844a0474 FAIL (2025-12-28 18:30:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ) context deadline exceeded ======LOGS====== 1m23.977394577s) [upload=486602]: Uploading segments... Dec 28 18:30:24.106371696 (555h1m23.977454741s) [upload=486602]: Got next segment. Inline: false Dec 28 18:30:24.106401292 (555h1m23.977484337s) [upload=486602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:30:24.106405650 (555h1m23.977488696s) [upload=486602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:30:24.356211154 (555h1m24.227294199s) [seg_pos={0 0}, upload=486602]: Begin upload segment... Dec 28 18:30:24.356307267 (555h1m24.227390302s) [seg_pos={0 0}, upload=486602]: Done begin upload segment. Dec 28 18:30:24.356344198 (555h1m24.227427243s) [noise=true , address=145.79.227.48:4301 , piece_id=KAUC6B4SNHGYVLVNJCKAXZ2ZOX23ORZJGN45PVCZ2FJ7WWA2DYNQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.362040965 (555h1m24.23312401s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.363959479 (555h1m24.235042514s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.365049704 (555h1m24.236132749s) [noise=true , address=145.79.227.37:4566 , piece_id=7JECM4YX64MUNTNVNBL3GDEOXBAJYTWTLI3AAQE27YBSIUJSZG6Q, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366012729 (555h1m24.237095774s) [noise=true , address=77.90.31.15:4697 , piece_id=B2HTKHQKFOOO6MRIDY33MNQDLR3QHWNIIFTXK4W44ZBFFJQHQWFA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366082111 (555h1m24.237165146s) [noise=true , address=77.90.31.14:4663 , piece_id=N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366707802 (555h1m24.237790847s) [noise=true , address=145.79.227.13:4803 , piece_id=WGD4HUTUX2OCQNIO3FNY4PR5AJEEJ7X2XM4PYTSZFKYDINNI44PA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.366805537 (555h1m24.237888582s) [noise=true , address=145.79.227.7:4295 , piece_id=764M3EY6NZQHJ2VNHYXP5QFGVCICQU6ACQO7KK7TKZFWCGLW7YZA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367179119 (555h1m24.238262164s) [noise=true , address=145.79.227.39:4015 , piece_id=KJDFSCM4R2QCQE4AXNMWZ3S2WDYXXCGX2AMEQTILBBOXKA4MLTNQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367317152 (555h1m24.238400187s) [noise=true , address=45.91.116.14:4192 , piece_id=SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367667419 (555h1m24.238750474s) [noise=true , address=145.79.227.38:4977 , piece_id=MPSYTE7G6SLY52KRFGM6I3ILMVITDYQHWCESTY53X6XEQJJWPDMQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.367832303 (555h1m24.238915338s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368185936 (555h1m24.239268981s) [noise=true , address=145.79.227.52:4447 , piece_id=5JBYMCC34PVNXD7QOAOKLJHYJBRAVGEHWJMSQCYLEOG4KJFJEDPA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368298510 (555h1m24.239381545s) [noise=true , address=145.79.227.15:4874 , piece_id=FT54ZA3ZN7NZ4LCNVJ2SPOBAYNQ6Q7IGVGRFRE7TJJDKXCPY7Q5A, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.368769107 (555h1m24.239852142s) [noise=true , address=145.79.227.41:4087 , piece_id=LKCKLONESBRTDKDVP4DA2JV3GHAETLT3ADIH7AQDNJZQRFJ65TYA, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:24.369292824 (555h1m24.240375869s) [noise=true , address=145.79.227.45:4231 , piece_id=LAPV3G65R5MDPAWY7FDCFTFJZY4CSEXMNJUF5PPMDG7GISMXSETQ, seg_pos={0 0}, upload=486602]: Uploading piece... Dec 28 18:30:25.987984716 (555h1m25.859067751s) [upload=486602]: Next returned nil segment Dec 28 18:30:25.987999474 (555h1m25.859082519s) [upload=486602]: Waiting for error group managing segments... Dec 28 18:30:26.371269827 (555h1m26.242352872s) [noise=true , address=145.79.227.39:4015 , piece_id=KJDFSCM4R2QCQE4AXNMWZ3S2WDYXXCGX2AMEQTILBBOXKA4MLTNQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.375273340 (555h1m26.246356375s) [noise=true , address=145.79.227.37:4566 , piece_id=7JECM4YX64MUNTNVNBL3GDEOXBAJYTWTLI3AAQE27YBSIUJSZG6Q, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.377276505 (555h1m26.24835955s) [noise=true , address=145.79.227.7:4295 , piece_id=764M3EY6NZQHJ2VNHYXP5QFGVCICQU6ACQO7KK7TKZFWCGLW7YZA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.385263112 (555h1m26.256346157s) [noise=true , address=145.79.227.38:4977 , piece_id=MPSYTE7G6SLY52KRFGM6I3ILMVITDYQHWCESTY53X6XEQJJWPDMQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.388294895 (555h1m26.25937794s) [noise=true , address=145.79.227.52:4447 , piece_id=5JBYMCC34PVNXD7QOAOKLJHYJBRAVGEHWJMSQCYLEOG4KJFJEDPA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.392247552 (555h1m26.263330597s) [noise=true , address=145.79.227.45:4231 , piece_id=LAPV3G65R5MDPAWY7FDCFTFJZY4CSEXMNJUF5PPMDG7GISMXSETQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.399676858 (555h1m26.270759894s) [noise=true , address=145.79.227.48:4301 , piece_id=KAUC6B4SNHGYVLVNJCKAXZ2ZOX23ORZJGN45PVCZ2FJ7WWA2DYNQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.400307971 (555h1m26.271391016s) [noise=true , address=145.79.227.41:4087 , piece_id=LKCKLONESBRTDKDVP4DA2JV3GHAETLT3ADIH7AQDNJZQRFJ65TYA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.402205133 (555h1m26.273288168s) [noise=true , address=145.79.227.15:4874 , piece_id=FT54ZA3ZN7NZ4LCNVJ2SPOBAYNQ6Q7IGVGRFRE7TJJDKXCPY7Q5A, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.565360970 (555h1m26.436444015s) [noise=true , address=77.90.31.14:4663 , piece_id=N336ELTV7WYFZGP6LSXGWQ4VA5FPZYZCOR6MYOIU7J3MVVIP5IRQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.589487802 (555h1m26.460570847s) [noise=true , address=145.79.227.13:4803 , piece_id=WGD4HUTUX2OCQNIO3FNY4PR5AJEEJ7X2XM4PYTSZFKYDINNI44PA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.630597672 (555h1m26.501680717s) [noise=true , address=45.91.116.14:4192 , piece_id=SPJHYU4QFIGCNOUXJRHC6PJ77NPAUX7PXKIRJPYRKQL2VEGOYRPQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.652758160 (555h1m26.523841205s) [noise=true , address=77.90.31.15:4697 , piece_id=B2HTKHQKFOOO6MRIDY33MNQDLR3QHWNIIFTXK4W44ZBFFJQHQWFA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: <nil> Dec 28 18:30:26.652783077 (555h1m26.523866113s) [seg_pos={0 0}, upload=486602]: Segment reached optimal threshold of 13 pieces. Dec 28 18:30:26.653003737 (555h1m26.524086782s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection; write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53738->145.79.227.3:4150: use of closed network connection Dec 28 18:30:26.653014498 (555h1m26.524097534s) [noise=true , address=145.79.227.3:4150 , piece_id=IXI6YYWC3VZDXB6GQIOF4PNAXEQXT4EQRZ3GXUREEUNPNTXHXHUQ, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653025519 (555h1m26.524108564s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57704->45.91.116.4:4940: use of closed network connection Dec 28 18:30:26.653033464 (555h1m26.524116509s) [noise=true , address=45.91.116.4:4940 , piece_id=IVFBCBSLTMG7OUQF7HEBQC4VK5PGI3GT5J42MZJP4NAOUF6E4DCA, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653216292 (555h1m26.524299337s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 18:30:26.653225009 (555h1m26.524308074s) [noise=true , address=45.91.116.10:4084 , piece_id=HP6KY3HIJWTJ2BGLYUMI6MQOTJQOZOATMMTLOEIIE7AIWLI6IUIA, seg_pos={0 0}, upload=486602]: Long tail cancellation due to optimal threshold Dec 28 18:30:26.653244837 (555h1m26.524327872s) [seg_pos={0 0}, upload=486602]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:30:26.653328276 (555h1m26.524411321s) [upload=486602]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:30:26.653334898 (555h1m26.524417943s) [upload=486602]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:30:26.957763122 (555h1m26.828846167s) [upload=486602]: Done uploading segments... Dec 28 18:30:26.957789813 (555h1m26.828872858s) [upload=486602]: Upload finished. err: <nil> Duration: 30.002771211s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X........X.............X.X........................X...] duration: 40.67328265s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:30:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 40.673314061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 3.371907608s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:30:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.371950841s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [....X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X..] duration: 5.97793625s trace: https://jaeger.dev.storj.io/trace/10cce5e50b12fdca FAIL (2025-12-28 18:29:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== context deadline exceeded ======LOGS====== Dec 28 18:28:18.510655659 (554h59m18.381738704s) [upload=486569]: Starting upload Dec 28 18:28:18.510739759 (554h59m18.381822794s) [upload=486569]: Done starting upload Dec 28 18:28:18.510866611 (554h59m18.381949646s) [upload=486569]: Uploading segments... Dec 28 18:28:18.510955830 (554h59m18.382038865s) [upload=486569]: Got next segment. Inline: false Dec 28 18:28:18.510997801 (554h59m18.382080846s) [upload=486569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:28:18.511003402 (554h59m18.382086447s) [upload=486569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:28:18.753609916 (554h59m18.624692951s) [seg_pos={0 0}, upload=486569]: Begin upload segment... Dec 28 18:28:18.753673928 (554h59m18.624756963s) [seg_pos={0 0}, upload=486569]: Done begin upload segment. Dec 28 18:28:18.753698926 (554h59m18.624781971s) [noise=true , address=145.79.227.21:4475 , piece_id=Z3MOXVOSAL3DD6HSQNEMROMTQW2VG4R4IFWNCNADJUGWA6TW4MDQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.753719335 (554h59m18.62480237s) [noise=true , address=45.91.116.9:4263 , piece_id=4OGTBR2P6KTS6OTBE5EFCWID624QSHJQ3WNJNJND4LBZQJC764OA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754103286 (554h59m18.625186321s) [noise=true , address=145.79.227.48:4301 , piece_id=UMEGJE2377E7WMTDSDJH6DKLVS4DICR7JFM7GZ7WKHMOF35R2MXQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754378079 (554h59m18.625461124s) [noise=true , address=45.91.116.5:4298 , piece_id=KWNNNBTDYPBDJNTGVXXFEKDZBTGRM26BMEW27LSQYCRFMVJ6FP7Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754787488 (554h59m18.625870524s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755053716 (554h59m18.626136751s) [noise=true , address=45.91.116.6:4975 , piece_id=C54VHUM3DO4OOVDCSICLR65VDMDVM7UWQ5DYEKFOV6W367ENML2A, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755319252 (554h59m18.626402287s) [noise=true , address=145.79.227.46:4603 , piece_id=EWGI4MV4KWI6ZPNIKG54GOJ35TM25LV6OKFWDW3EMJH5T4OS7X6Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755587071 (554h59m18.626670106s) [noise=true , address=145.79.227.15:4876 , piece_id=SIKUFW7EZSB5JPYAVVTOOCTOUUBFYZXTUEQBC2AK3GBAVUE3BTNQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755909364 (554h59m18.6269924s) [noise=true , address=45.91.116.12:4227 , piece_id=HH6FCJIS5OSJDVLNYY5PQJKT4GEVQOVSWU7FQFRG5HOQWVJAQNSQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.757152271 (554h59m18.628235317s) [noise=true , address=145.79.227.51:4409 , piece_id=ZXRRW6NXDE2MGQ7N7QHLBL7XLW3BRBCO4ZBPC7TCU253MCYSHRRA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.758317020 (554h59m18.629400065s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.759644397 (554h59m18.630727432s) [noise=true , address=145.79.227.41:4087 , piece_id=UJ3KJP327STFFFSM2O2DETOYO5ZYMD4PCSPAS7WB4SKWJKAUJRNQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.761127371 (554h59m18.632210406s) [noise=true , address=145.79.227.11:4732 , piece_id=OLQVBQED5QTKCMRFENXBTQVOXGLBEFJVAOMOITS7MPELN3JAVTVQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.762709013 (554h59m18.633792058s) [noise=true , address=77.90.31.12:4590 , piece_id=IRIW2ZL2JGWIVKWI4ORBZUGXEYSGLKAZQSPNAUNK76WK6QMTTUOQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.763946980 (554h59m18.635030015s) [noise=true , address=77.90.31.15:4697 , piece_id=2ZT44C2RCN4F7LXIHTJPOBZJRKJ5XAX7HOPVNRVCQJXPGTCJRB6Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.764949348 (554h59m18.636032394s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:20.469372967 (554h59m20.340456002s) [upload=486569]: Next returned nil segment Dec 28 18:28:20.469387253 (554h59m20.340470289s) [upload=486569]: Waiting for error group managing segments... Dec 28 18:28:20.766232138 (554h59m20.637315173s) [noise=true , address=145.79.227.11:4732 , piece_id=OLQVBQED5QTKCMRFENXBTQVOXGLBEFJVAOMOITS7MPELN3JAVTVQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.776862000 (554h59m20.647945035s) [noise=true , address=145.79.227.48:4301 , piece_id=UMEGJE2377E7WMTDSDJH6DKLVS4DICR7JFM7GZ7WKHMOF35R2MXQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.777963156 (554h59m20.649046191s) [noise=true , address=145.79.227.21:4475 , piece_id=Z3MOXVOSAL3DD6HSQNEMROMTQW2VG4R4IFWNCNADJUGWA6TW4MDQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.778884992 (554h59m20.649968027s) [noise=true , address=145.79.227.46:4603 , piece_id=EWGI4MV4KWI6ZPNIKG54GOJ35TM25LV6OKFWDW3EMJH5T4OS7X6Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.781995444 (554h59m20.653078479s) [noise=true , address=145.79.227.15:4876 , piece_id=SIKUFW7EZSB5JPYAVVTOOCTOUUBFYZXTUEQBC2AK3GBAVUE3BTNQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.786417855 (554h59m20.65750089s) [noise=true , address=145.79.227.51:4409 , piece_id=ZXRRW6NXDE2MGQ7N7QHLBL7XLW3BRBCO4ZBPC7TCU253MCYSHRRA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.794969359 (554h59m20.666052394s) [noise=true , address=145.79.227.41:4087 , piece_id=UJ3KJP327STFFFSM2O2DETOYO5ZYMD4PCSPAS7WB4SKWJKAUJRNQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.976239159 (554h59m20.847322194s) [noise=true , address=77.90.31.12:4590 , piece_id=IRIW2ZL2JGWIVKWI4ORBZUGXEYSGLKAZQSPNAUNK76WK6QMTTUOQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.000438899 (554h59m20.871521954s) [noise=true , address=45.91.116.6:4975 , piece_id=C54VHUM3DO4OOVDCSICLR65VDMDVM7UWQ5DYEKFOV6W367ENML2A, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.027118281 (554h59m20.898201316s) [noise=true , address=45.91.116.12:4227 , piece_id=HH6FCJIS5OSJDVLNYY5PQJKT4GEVQOVSWU7FQFRG5HOQWVJAQNSQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.063174649 (554h59m20.934257684s) [noise=true , address=77.90.31.15:4697 , piece_id=2ZT44C2RCN4F7LXIHTJPOBZJRKJ5XAX7HOPVNRVCQJXPGTCJRB6Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.167129967 (554h59m21.038213002s) [noise=true , address=45.91.116.5:4298 , piece_id=KWNNNBTDYPBDJNTGVXXFEKDZBTGRM26BMEW27LSQYCRFMVJ6FP7Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.739100729 (554h59m21.610183774s) [noise=true , address=45.91.116.9:4263 , piece_id=4OGTBR2P6KTS6OTBE5EFCWID624QSHJQ3WNJNJND4LBZQJC764OA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.739127150 (554h59m21.610210185s) [seg_pos={0 0}, upload=486569]: Segment reached optimal threshold of 13 pieces. Dec 28 18:28:21.739370012 (554h59m21.610453047s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739378138 (554h59m21.610461173s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739603837 (554h59m21.610686872s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739611793 (554h59m21.610694838s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739663872 (554h59m21.610746907s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739682357 (554h59m21.610765402s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739723716 (554h59m21.610806761s) [seg_pos={0 0}, upload=486569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:28:21.739847220 (554h59m21.610930265s) [upload=486569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:28:21.739853894 (554h59m21.610936939s) [upload=486569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:28:22.032616163 (554h59m21.903699208s) [upload=486569]: Done uploading segments... Dec 28 18:28:22.032643936 (554h59m21.903727001s) [upload=486569]: Upload finished. err: <nil> Duration: 5.977965746s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X........X.............X.X........................X..] duration: 44.881749856s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:29:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 44.88177832s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.....X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X.] duration: 14.503311376s trace: https://jaeger.dev.storj.io/trace/10cce5e50b12fdca FAIL (2025-12-28 18:29:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== context deadline exceeded ======LOGS====== Dec 28 18:28:18.510655659 (554h59m18.381738704s) [upload=486569]: Starting upload Dec 28 18:28:18.510739759 (554h59m18.381822794s) [upload=486569]: Done starting upload Dec 28 18:28:18.510866611 (554h59m18.381949646s) [upload=486569]: Uploading segments... Dec 28 18:28:18.510955830 (554h59m18.382038865s) [upload=486569]: Got next segment. Inline: false Dec 28 18:28:18.510997801 (554h59m18.382080846s) [upload=486569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:28:18.511003402 (554h59m18.382086447s) [upload=486569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:28:18.753609916 (554h59m18.624692951s) [seg_pos={0 0}, upload=486569]: Begin upload segment... Dec 28 18:28:18.753673928 (554h59m18.624756963s) [seg_pos={0 0}, upload=486569]: Done begin upload segment. Dec 28 18:28:18.753698926 (554h59m18.624781971s) [noise=true , address=145.79.227.21:4475 , piece_id=Z3MOXVOSAL3DD6HSQNEMROMTQW2VG4R4IFWNCNADJUGWA6TW4MDQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.753719335 (554h59m18.62480237s) [noise=true , address=45.91.116.9:4263 , piece_id=4OGTBR2P6KTS6OTBE5EFCWID624QSHJQ3WNJNJND4LBZQJC764OA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754103286 (554h59m18.625186321s) [noise=true , address=145.79.227.48:4301 , piece_id=UMEGJE2377E7WMTDSDJH6DKLVS4DICR7JFM7GZ7WKHMOF35R2MXQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754378079 (554h59m18.625461124s) [noise=true , address=45.91.116.5:4298 , piece_id=KWNNNBTDYPBDJNTGVXXFEKDZBTGRM26BMEW27LSQYCRFMVJ6FP7Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754787488 (554h59m18.625870524s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755053716 (554h59m18.626136751s) [noise=true , address=45.91.116.6:4975 , piece_id=C54VHUM3DO4OOVDCSICLR65VDMDVM7UWQ5DYEKFOV6W367ENML2A, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755319252 (554h59m18.626402287s) [noise=true , address=145.79.227.46:4603 , piece_id=EWGI4MV4KWI6ZPNIKG54GOJ35TM25LV6OKFWDW3EMJH5T4OS7X6Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755587071 (554h59m18.626670106s) [noise=true , address=145.79.227.15:4876 , piece_id=SIKUFW7EZSB5JPYAVVTOOCTOUUBFYZXTUEQBC2AK3GBAVUE3BTNQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755909364 (554h59m18.6269924s) [noise=true , address=45.91.116.12:4227 , piece_id=HH6FCJIS5OSJDVLNYY5PQJKT4GEVQOVSWU7FQFRG5HOQWVJAQNSQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.757152271 (554h59m18.628235317s) [noise=true , address=145.79.227.51:4409 , piece_id=ZXRRW6NXDE2MGQ7N7QHLBL7XLW3BRBCO4ZBPC7TCU253MCYSHRRA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.758317020 (554h59m18.629400065s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.759644397 (554h59m18.630727432s) [noise=true , address=145.79.227.41:4087 , piece_id=UJ3KJP327STFFFSM2O2DETOYO5ZYMD4PCSPAS7WB4SKWJKAUJRNQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.761127371 (554h59m18.632210406s) [noise=true , address=145.79.227.11:4732 , piece_id=OLQVBQED5QTKCMRFENXBTQVOXGLBEFJVAOMOITS7MPELN3JAVTVQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.762709013 (554h59m18.633792058s) [noise=true , address=77.90.31.12:4590 , piece_id=IRIW2ZL2JGWIVKWI4ORBZUGXEYSGLKAZQSPNAUNK76WK6QMTTUOQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.763946980 (554h59m18.635030015s) [noise=true , address=77.90.31.15:4697 , piece_id=2ZT44C2RCN4F7LXIHTJPOBZJRKJ5XAX7HOPVNRVCQJXPGTCJRB6Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.764949348 (554h59m18.636032394s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:20.469372967 (554h59m20.340456002s) [upload=486569]: Next returned nil segment Dec 28 18:28:20.469387253 (554h59m20.340470289s) [upload=486569]: Waiting for error group managing segments... Dec 28 18:28:20.766232138 (554h59m20.637315173s) [noise=true , address=145.79.227.11:4732 , piece_id=OLQVBQED5QTKCMRFENXBTQVOXGLBEFJVAOMOITS7MPELN3JAVTVQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.776862000 (554h59m20.647945035s) [noise=true , address=145.79.227.48:4301 , piece_id=UMEGJE2377E7WMTDSDJH6DKLVS4DICR7JFM7GZ7WKHMOF35R2MXQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.777963156 (554h59m20.649046191s) [noise=true , address=145.79.227.21:4475 , piece_id=Z3MOXVOSAL3DD6HSQNEMROMTQW2VG4R4IFWNCNADJUGWA6TW4MDQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.778884992 (554h59m20.649968027s) [noise=true , address=145.79.227.46:4603 , piece_id=EWGI4MV4KWI6ZPNIKG54GOJ35TM25LV6OKFWDW3EMJH5T4OS7X6Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.781995444 (554h59m20.653078479s) [noise=true , address=145.79.227.15:4876 , piece_id=SIKUFW7EZSB5JPYAVVTOOCTOUUBFYZXTUEQBC2AK3GBAVUE3BTNQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.786417855 (554h59m20.65750089s) [noise=true , address=145.79.227.51:4409 , piece_id=ZXRRW6NXDE2MGQ7N7QHLBL7XLW3BRBCO4ZBPC7TCU253MCYSHRRA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.794969359 (554h59m20.666052394s) [noise=true , address=145.79.227.41:4087 , piece_id=UJ3KJP327STFFFSM2O2DETOYO5ZYMD4PCSPAS7WB4SKWJKAUJRNQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.976239159 (554h59m20.847322194s) [noise=true , address=77.90.31.12:4590 , piece_id=IRIW2ZL2JGWIVKWI4ORBZUGXEYSGLKAZQSPNAUNK76WK6QMTTUOQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.000438899 (554h59m20.871521954s) [noise=true , address=45.91.116.6:4975 , piece_id=C54VHUM3DO4OOVDCSICLR65VDMDVM7UWQ5DYEKFOV6W367ENML2A, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.027118281 (554h59m20.898201316s) [noise=true , address=45.91.116.12:4227 , piece_id=HH6FCJIS5OSJDVLNYY5PQJKT4GEVQOVSWU7FQFRG5HOQWVJAQNSQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.063174649 (554h59m20.934257684s) [noise=true , address=77.90.31.15:4697 , piece_id=2ZT44C2RCN4F7LXIHTJPOBZJRKJ5XAX7HOPVNRVCQJXPGTCJRB6Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.167129967 (554h59m21.038213002s) [noise=true , address=45.91.116.5:4298 , piece_id=KWNNNBTDYPBDJNTGVXXFEKDZBTGRM26BMEW27LSQYCRFMVJ6FP7Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.739100729 (554h59m21.610183774s) [noise=true , address=45.91.116.9:4263 , piece_id=4OGTBR2P6KTS6OTBE5EFCWID624QSHJQ3WNJNJND4LBZQJC764OA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.739127150 (554h59m21.610210185s) [seg_pos={0 0}, upload=486569]: Segment reached optimal threshold of 13 pieces. Dec 28 18:28:21.739370012 (554h59m21.610453047s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739378138 (554h59m21.610461173s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739603837 (554h59m21.610686872s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739611793 (554h59m21.610694838s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739663872 (554h59m21.610746907s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739682357 (554h59m21.610765402s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739723716 (554h59m21.610806761s) [seg_pos={0 0}, upload=486569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:28:21.739847220 (554h59m21.610930265s) [upload=486569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:28:21.739853894 (554h59m21.610936939s) [upload=486569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:28:22.032616163 (554h59m21.903699208s) [upload=486569]: Done uploading segments... Dec 28 18:28:22.032643936 (554h59m21.903727001s) [upload=486569]: Upload finished. err: <nil> Duration: 14.503341233s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 3.652190256s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:28:59 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.652223338s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..X] duration: 30.001735681s trace: https://jaeger.dev.storj.io/trace/10cce5e50b12fdca FAIL (2025-12-28 18:28:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== context deadline exceeded ======LOGS====== Dec 28 18:28:18.510655659 (554h59m18.381738704s) [upload=486569]: Starting upload Dec 28 18:28:18.510739759 (554h59m18.381822794s) [upload=486569]: Done starting upload Dec 28 18:28:18.510866611 (554h59m18.381949646s) [upload=486569]: Uploading segments... Dec 28 18:28:18.510955830 (554h59m18.382038865s) [upload=486569]: Got next segment. Inline: false Dec 28 18:28:18.510997801 (554h59m18.382080846s) [upload=486569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:28:18.511003402 (554h59m18.382086447s) [upload=486569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:28:18.753609916 (554h59m18.624692951s) [seg_pos={0 0}, upload=486569]: Begin upload segment... Dec 28 18:28:18.753673928 (554h59m18.624756963s) [seg_pos={0 0}, upload=486569]: Done begin upload segment. Dec 28 18:28:18.753698926 (554h59m18.624781971s) [noise=true , address=145.79.227.21:4475 , piece_id=Z3MOXVOSAL3DD6HSQNEMROMTQW2VG4R4IFWNCNADJUGWA6TW4MDQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.753719335 (554h59m18.62480237s) [noise=true , address=45.91.116.9:4263 , piece_id=4OGTBR2P6KTS6OTBE5EFCWID624QSHJQ3WNJNJND4LBZQJC764OA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754103286 (554h59m18.625186321s) [noise=true , address=145.79.227.48:4301 , piece_id=UMEGJE2377E7WMTDSDJH6DKLVS4DICR7JFM7GZ7WKHMOF35R2MXQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754378079 (554h59m18.625461124s) [noise=true , address=45.91.116.5:4298 , piece_id=KWNNNBTDYPBDJNTGVXXFEKDZBTGRM26BMEW27LSQYCRFMVJ6FP7Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.754787488 (554h59m18.625870524s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755053716 (554h59m18.626136751s) [noise=true , address=45.91.116.6:4975 , piece_id=C54VHUM3DO4OOVDCSICLR65VDMDVM7UWQ5DYEKFOV6W367ENML2A, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755319252 (554h59m18.626402287s) [noise=true , address=145.79.227.46:4603 , piece_id=EWGI4MV4KWI6ZPNIKG54GOJ35TM25LV6OKFWDW3EMJH5T4OS7X6Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755587071 (554h59m18.626670106s) [noise=true , address=145.79.227.15:4876 , piece_id=SIKUFW7EZSB5JPYAVVTOOCTOUUBFYZXTUEQBC2AK3GBAVUE3BTNQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.755909364 (554h59m18.6269924s) [noise=true , address=45.91.116.12:4227 , piece_id=HH6FCJIS5OSJDVLNYY5PQJKT4GEVQOVSWU7FQFRG5HOQWVJAQNSQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.757152271 (554h59m18.628235317s) [noise=true , address=145.79.227.51:4409 , piece_id=ZXRRW6NXDE2MGQ7N7QHLBL7XLW3BRBCO4ZBPC7TCU253MCYSHRRA, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.758317020 (554h59m18.629400065s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.759644397 (554h59m18.630727432s) [noise=true , address=145.79.227.41:4087 , piece_id=UJ3KJP327STFFFSM2O2DETOYO5ZYMD4PCSPAS7WB4SKWJKAUJRNQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.761127371 (554h59m18.632210406s) [noise=true , address=145.79.227.11:4732 , piece_id=OLQVBQED5QTKCMRFENXBTQVOXGLBEFJVAOMOITS7MPELN3JAVTVQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.762709013 (554h59m18.633792058s) [noise=true , address=77.90.31.12:4590 , piece_id=IRIW2ZL2JGWIVKWI4ORBZUGXEYSGLKAZQSPNAUNK76WK6QMTTUOQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.763946980 (554h59m18.635030015s) [noise=true , address=77.90.31.15:4697 , piece_id=2ZT44C2RCN4F7LXIHTJPOBZJRKJ5XAX7HOPVNRVCQJXPGTCJRB6Q, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:18.764949348 (554h59m18.636032394s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Uploading piece... Dec 28 18:28:20.469372967 (554h59m20.340456002s) [upload=486569]: Next returned nil segment Dec 28 18:28:20.469387253 (554h59m20.340470289s) [upload=486569]: Waiting for error group managing segments... Dec 28 18:28:20.766232138 (554h59m20.637315173s) [noise=true , address=145.79.227.11:4732 , piece_id=OLQVBQED5QTKCMRFENXBTQVOXGLBEFJVAOMOITS7MPELN3JAVTVQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.776862000 (554h59m20.647945035s) [noise=true , address=145.79.227.48:4301 , piece_id=UMEGJE2377E7WMTDSDJH6DKLVS4DICR7JFM7GZ7WKHMOF35R2MXQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.777963156 (554h59m20.649046191s) [noise=true , address=145.79.227.21:4475 , piece_id=Z3MOXVOSAL3DD6HSQNEMROMTQW2VG4R4IFWNCNADJUGWA6TW4MDQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.778884992 (554h59m20.649968027s) [noise=true , address=145.79.227.46:4603 , piece_id=EWGI4MV4KWI6ZPNIKG54GOJ35TM25LV6OKFWDW3EMJH5T4OS7X6Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.781995444 (554h59m20.653078479s) [noise=true , address=145.79.227.15:4876 , piece_id=SIKUFW7EZSB5JPYAVVTOOCTOUUBFYZXTUEQBC2AK3GBAVUE3BTNQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.786417855 (554h59m20.65750089s) [noise=true , address=145.79.227.51:4409 , piece_id=ZXRRW6NXDE2MGQ7N7QHLBL7XLW3BRBCO4ZBPC7TCU253MCYSHRRA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.794969359 (554h59m20.666052394s) [noise=true , address=145.79.227.41:4087 , piece_id=UJ3KJP327STFFFSM2O2DETOYO5ZYMD4PCSPAS7WB4SKWJKAUJRNQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:20.976239159 (554h59m20.847322194s) [noise=true , address=77.90.31.12:4590 , piece_id=IRIW2ZL2JGWIVKWI4ORBZUGXEYSGLKAZQSPNAUNK76WK6QMTTUOQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.000438899 (554h59m20.871521954s) [noise=true , address=45.91.116.6:4975 , piece_id=C54VHUM3DO4OOVDCSICLR65VDMDVM7UWQ5DYEKFOV6W367ENML2A, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.027118281 (554h59m20.898201316s) [noise=true , address=45.91.116.12:4227 , piece_id=HH6FCJIS5OSJDVLNYY5PQJKT4GEVQOVSWU7FQFRG5HOQWVJAQNSQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.063174649 (554h59m20.934257684s) [noise=true , address=77.90.31.15:4697 , piece_id=2ZT44C2RCN4F7LXIHTJPOBZJRKJ5XAX7HOPVNRVCQJXPGTCJRB6Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.167129967 (554h59m21.038213002s) [noise=true , address=45.91.116.5:4298 , piece_id=KWNNNBTDYPBDJNTGVXXFEKDZBTGRM26BMEW27LSQYCRFMVJ6FP7Q, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.739100729 (554h59m21.610183774s) [noise=true , address=45.91.116.9:4263 , piece_id=4OGTBR2P6KTS6OTBE5EFCWID624QSHJQ3WNJNJND4LBZQJC764OA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: <nil> Dec 28 18:28:21.739127150 (554h59m21.610210185s) [seg_pos={0 0}, upload=486569]: Segment reached optimal threshold of 13 pieces. Dec 28 18:28:21.739370012 (554h59m21.610453047s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739378138 (554h59m21.610461173s) [noise=true , address=145.79.227.43:4157 , piece_id=DGTK4F7D37ZGGOSWZUT67KRIBJ4G3ZL3FOPNCELGHPP2KL3QP3PQ, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739603837 (554h59m21.610686872s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739611793 (554h59m21.610694838s) [noise=true , address=145.79.227.35:4482 , piece_id=OLDUYIJ33O2UP7ABG56BEKPX3G76KDU3ZTVMAU4VYABAPGPFVTUA, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739663872 (554h59m21.610746907s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 18:28:21.739682357 (554h59m21.610765402s) [noise=true , address=145.79.227.34:4907 , piece_id=HAE4QTFFUAOJWZXFPJLSQKNDNILZB5H3T5AKRFGM7YSBEG7LYLWQ, seg_pos={0 0}, upload=486569]: Long tail cancellation due to optimal threshold Dec 28 18:28:21.739723716 (554h59m21.610806761s) [seg_pos={0 0}, upload=486569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:28:21.739847220 (554h59m21.610930265s) [upload=486569]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:28:21.739853894 (554h59m21.610936939s) [upload=486569]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:28:22.032616163 (554h59m21.903699208s) [upload=486569]: Done uploading segments... Dec 28 18:28:22.032643936 (554h59m21.903727001s) [upload=486569]: Upload finished. err: <nil> Duration: 30.00175672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X........X.............X.X........................X.] duration: 34.67521182s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:28:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 34.675239853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 3.092048529s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:27:54 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.092064629s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X........X.............X.X........................X] duration: 1m0.003715545s trace: https://jaeger.dev.storj.io/trace/9c4c1479ce693a4 FAIL (2025-12-28 18:27:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== =true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.493626076 (554h57m48.364709112s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494053550 (554h57m48.365136595s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494372728 (554h57m48.365455763s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.494727603 (554h57m48.365810648s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495038165 (554h57m48.3661212s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.495548747 (554h57m48.366631792s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496002832 (554h57m48.367085877s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.496472646 (554h57m48.367555691s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.497024507 (554h57m48.368107552s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498108050 (554h57m48.369191095s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.498736827 (554h57m48.369819872s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.499362980 (554h57m48.370446015s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500528039 (554h57m48.371611084s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.500739191 (554h57m48.371822246s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:48.502540641 (554h57m48.373623686s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Uploading piece... Dec 28 18:26:49.886286264 (554h57m49.757369299s) [upload=486545]: Next returned nil segment Dec 28 18:26:49.886299729 (554h57m49.757382765s) [upload=486545]: Waiting for error group managing segments... Dec 28 18:26:50.220246773 (554h57m50.091329808s) [noise=true , address=64.49.13.27:4968 , piece_id=HT2PFH22PPB4IWMJALXPPDEX3L3KMKBTFR2TAQHKFOPXFMWSYUBQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.389758503 (554h57m50.260841538s) [noise=true , address=145.79.227.15:4892 , piece_id=M6334AKJCIU7X5AZEBHROGD75UQO7DQFJF2VHUHV5LIZ7KCSCJSA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.402131182 (554h57m50.273214218s) [noise=true , address=145.79.227.46:4619 , piece_id=5XJEFH6LAFWU2C6RYWOSPRF6UKTYCXMWY4XWIR7G32LE7L6UXNIA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.615224770 (554h57m50.486307805s) [noise=true , address=77.90.31.12:4612 , piece_id=FTEH4UYQKG7SEJIUCUTENWU7U6LYEK5HMZZGEKRZUYF2MU4V5XYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.627993474 (554h57m50.499076519s) [noise=true , address=77.90.31.13:4654 , piece_id=WEJGYGJ7O2ZIROPJFYS7KHL4ZWEMVDADM6WFSLE72MF4UQTYVL5A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.643120118 (554h57m50.514203163s) [noise=true , address=77.90.31.21:4891 , piece_id=CB3LSDMDPR27MTQP3IBVSME5PVWS2RPA236NZ3K4LLYZDAWEUUYA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:50.985393065 (554h57m50.8564761s) [noise=true , address=77.90.31.18:4835 , piece_id=WTDLS4TMLI2U3AILEMZQTGOJ2UQQU6SEAMV3QR736RORIVHEWJJA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.258403203 (554h57m51.129486238s) [noise=true , address=77.90.31.17:4803 , piece_id=BME6GMCFVKOWPPIAAPYJ4HXCXSWYPSRII3PL3DATJCWI2I76O6FA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.300313959 (554h57m51.171397004s) [noise=true , address=5.231.73.4:4032 , piece_id=T6AQP2AVIAJ2PTENJT5OBVPWT2MID5RTC6YLB74OVXF4BNVJ5MMA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.345167979 (554h57m51.216251044s) [noise=true , address=145.79.227.44:4209 , piece_id=KPOBJ3HIHRT7DKSVMVTG46MITDOAVPDV3SUII6T5OCZNWCVIRPXQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.347758331 (554h57m51.218841366s) [noise=true , address=145.79.227.30:4787 , piece_id=C2ZQXREAOG6GYHMIOSXN3HFX63KDJJT7TKYY5MOURZXCZYKDFK2Q, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.352068438 (554h57m51.223151473s) [noise=true , address=64.49.13.154:4928 , piece_id=WZ6IMCSRQLEPDLKKNDXAXNEJL5LWXGFTMKRYVS6MVGYXQA3ESZ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.392273756 (554h57m51.263356791s) [noise=true , address=145.79.227.35:4493 , piece_id=Q5G7QXDEC3OIXA3L4HLKOFIRRFMW5UUFGKF4T5TQPEPOD7GQGNVA, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404848671 (554h57m51.275931706s) [noise=true , address=145.79.227.40:4057 , piece_id=KJCWDBZYVCCE47ZKMK4PJAIPHFTARIHSE6ETFVTY2KD6EXI33PGQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: <nil> Dec 28 18:26:51.404873238 (554h57m51.275956273s) [seg_pos={0 0}, upload=486545]: Segment reached optimal threshold of 14 pieces. Dec 28 18:26:51.405254233 (554h57m51.276337268s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ezboNAWPbaK9UDENYoBb3iHjNU2ahn7mikiteN7y3Aj6uw4Nx): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405271787 (554h57m51.276354822s) [noise=true , address=145.79.227.49:4348 , piece_id=OXWSFRIUCDEINMRYKBRXGSQDKP3UM7T75MRUGXZZ3HB24XG7LXZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405320129 (554h57m51.276403174s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405329797 (554h57m51.276412842s) [noise=true , address=145.79.227.43:4171 , piece_id=SG6TTXNF6OGRHZ2VKFCLB2YCM4N7CRW3RDHTOLPUAK6272YEXQ3A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405482058 (554h57m51.276565093s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CqtCR6R1o8wyteG2sLVScundbtjyEgouMLY64KTeoqFHwRtey): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:51.405489101 (554h57m51.276572156s) [noise=true , address=45.91.116.13:4168 , piece_id=KFRJ3VQ25WNXKUWQCGELCXCJAQX2LVXR4ZZWUFVGYTF3A54I3Z2A, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405595594 (554h57m51.276678639s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC): protocol: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55446->145.79.227.17:4979: use of closed network connection Dec 28 18:26:51.405605482 (554h57m51.276688527s) [noise=true , address=145.79.227.17:4979 , piece_id=IR6Z5AYUWZ6LVT2JEQ7RXRTKOI6Z4TBAR4FSATG4WKOJNY4RLZZQ, seg_pos={0 0}, upload=486545]: Long tail cancellation due to optimal threshold Dec 28 18:26:51.405626783 (554h57m51.276709818s) [seg_pos={0 0}, upload=486545]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:26:51.405706044 (554h57m51.276789079s) [upload=486545]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:51.405710612 (554h57m51.276793647s) [upload=486545]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:51.705670608 (554h57m51.576753673s) [upload=486545]: Done uploading segments... Dec 28 18:26:51.705694163 (554h57m51.576777208s) [upload=486545]: Upload finished. err: <nil> Duration: 1m0.003736233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X..] duration: 5.837119419s trace: https://jaeger.dev.storj.io/trace/3e86b8dbc57740e4 FAIL (2025-12-28 18:27:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: CBZGUSSQCQO6MSQTQBX2QCE6YDT5TZYPPL7H5PQDHWTBBDNYAO3A) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: NGSRRP5YODM3SFOYXFOJU472B7QQHZP53CNQBTVBCH3SD4AE63JQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NUCPTHPSFAAP57LUOOLKALYTE6D5MEYSQXDMO4UPQX6WLVI3CI5A) context deadline exceeded ======LOGS====== 8 18:26:23.223132473 (554h57m23.094215518s) [upload=486540]: Starting upload Dec 28 18:26:23.223206554 (554h57m23.094289619s) [upload=486540]: Done starting upload Dec 28 18:26:23.223392588 (554h57m23.094475633s) [upload=486540]: Uploading segments... Dec 28 18:26:23.223430210 (554h57m23.094513245s) [upload=486540]: Got next segment. Inline: false Dec 28 18:26:23.223458774 (554h57m23.094541819s) [upload=486540]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:26:23.223463824 (554h57m23.094546859s) [upload=486540]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:26:23.466996332 (554h57m23.338079367s) [seg_pos={0 0}, upload=486540]: Begin upload segment... Dec 28 18:26:23.467052268 (554h57m23.338135303s) [seg_pos={0 0}, upload=486540]: Done begin upload segment. Dec 28 18:26:23.467128193 (554h57m23.338211229s) [noise=true , address=145.79.227.50:4373 , piece_id=SBKGFELBFZJUUUP22OM4GLNV5FXPTO4GKWTFKE7NEVQXDIA44JSA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.467137471 (554h57m23.338220516s) [noise=true , address=145.79.227.32:4834 , piece_id=7WP7UODNJ6DPVN6VLHF5PGATODA276DUFLNUJKOD7JY5VGCMOTXA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.467766068 (554h57m23.338849103s) [noise=true , address=145.79.227.9:4366 , piece_id=J5WM6C3LUXXNTNX5APTY3GEONXUMZ5ZPOX3YZLPWCHLRJNPJSZ6Q, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.468245401 (554h57m23.339328446s) [noise=true , address=45.91.116.6:4975 , piece_id=REHMBIHT2FLO223AJ6ILW27FVGE3U3O3TVIZPM7I65SHWMIPCP7Q, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.469190711 (554h57m23.340273766s) [noise=true , address=145.79.227.8:4329 , piece_id=GJ74JSEIN5VQZLON3BBHJF3VJZ7JT23OGDPPGORMIT7ZKTSMIVLQ, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.469979403 (554h57m23.341062448s) [noise=true , address=45.91.116.11:4118 , piece_id=LPSKJYE477KZZF4EZ5UA7YWSUTZ3RZEHF54VBR2RD7D3FVXXKBDA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.471440094 (554h57m23.342523139s) [noise=true , address=145.79.227.21:4473 , piece_id=U7VUTPIX6VEM3IFCEHSIRXVLD2SIDHR4VWSZG2IKWUFBXY6J4FCA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.471872908 (554h57m23.342955943s) [noise=true , address=77.90.31.14:4661 , piece_id=DHN6UF3YP74GS2GC4L6ZMTIKDVDDBMCERIH4EJ46GMMBEMRZ4FPA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.472871359 (554h57m23.343954405s) [noise=true , address=45.91.116.5:4297 , piece_id=NUCPTHPSFAAP57LUOOLKALYTE6D5MEYSQXDMO4UPQX6WLVI3CI5A, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.474102874 (554h57m23.345185919s) [noise=true , address=145.79.227.16:4911 , piece_id=6G4JXFD6OOM47PKD3UYXDIYWX6RJ3V35ASKVVF3AFJPZJ7HUAX7A, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.477653535 (554h57m23.34873657s) [noise=true , address=45.91.116.10:4083 , piece_id=CBZGUSSQCQO6MSQTQBX2QCE6YDT5TZYPPL7H5PQDHWTBBDNYAO3A, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.484617305 (554h57m23.35570035s) [noise=true , address=145.79.227.11:4731 , piece_id=N6OFZ3RG4O64GUH75ANLC42UP5XYDDR7SPC6DK7AGSOIPMIR3JGQ, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.486794101 (554h57m23.357877136s) [noise=true , address=145.79.227.42:4121 , piece_id=IKX3O3K2EQSOUEKEDYI4UDOMFG3LPNHC7BNFPDFD4CEUSD6Z4BIA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.488215697 (554h57m23.359298742s) [noise=true , address=145.79.227.3:4150 , piece_id=5J7TDN4L3RGQLKI27FOD4WKNPVOXZFYZCY4APIBXFXXXKYRXQ4YA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.489318788 (554h57m23.360401823s) [noise=true , address=45.91.116.14:4191 , piece_id=NGSRRP5YODM3SFOYXFOJU472B7QQHZP53CNQBTVBCH3SD4AE63JQ, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.490544642 (554h57m23.361627677s) [noise=true , address=145.79.227.19:4077 , piece_id=O3LJG77WZHLCAFQR73KAKV5YCIQKKEMHDB5SYNZUQW4PDTUOXUVA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:25.196094737 (554h57m25.067177772s) [upload=486540]: Next returned nil segment Dec 28 18:26:25.196119805 (554h57m25.06720284s) [upload=486540]: Waiting for error group managing segments... Dec 28 18:26:25.484279761 (554h57m25.355362796s) [noise=true , address=145.79.227.21:4473 , piece_id=U7VUTPIX6VEM3IFCEHSIRXVLD2SIDHR4VWSZG2IKWUFBXY6J4FCA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.486082404 (554h57m25.357165439s) [noise=true , address=145.79.227.9:4366 , piece_id=J5WM6C3LUXXNTNX5APTY3GEONXUMZ5ZPOX3YZLPWCHLRJNPJSZ6Q, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.488101639 (554h57m25.359184684s) [noise=true , address=145.79.227.8:4329 , piece_id=GJ74JSEIN5VQZLON3BBHJF3VJZ7JT23OGDPPGORMIT7ZKTSMIVLQ, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.488490249 (554h57m25.359573294s) [noise=true , address=145.79.227.32:4834 , piece_id=7WP7UODNJ6DPVN6VLHF5PGATODA276DUFLNUJKOD7JY5VGCMOTXA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.493244001 (554h57m25.364327036s) [noise=true , address=145.79.227.19:4077 , piece_id=O3LJG77WZHLCAFQR73KAKV5YCIQKKEMHDB5SYNZUQW4PDTUOXUVA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.495432689 (554h57m25.366515734s) [noise=true , address=145.79.227.3:4150 , piece_id=5J7TDN4L3RGQLKI27FOD4WKNPVOXZFYZCY4APIBXFXXXKYRXQ4YA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.503725950 (554h57m25.374808995s) [noise=true , address=145.79.227.50:4373 , piece_id=SBKGFELBFZJUUUP22OM4GLNV5FXPTO4GKWTFKE7NEVQXDIA44JSA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.510661887 (554h57m25.381744932s) [noise=true , address=145.79.227.11:4731 , piece_id=N6OFZ3RG4O64GUH75ANLC42UP5XYDDR7SPC6DK7AGSOIPMIR3JGQ, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.517550905 (554h57m25.38863395s) [noise=true , address=145.79.227.42:4121 , piece_id=IKX3O3K2EQSOUEKEDYI4UDOMFG3LPNHC7BNFPDFD4CEUSD6Z4BIA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.545790037 (554h57m25.416873072s) [noise=true , address=145.79.227.16:4911 , piece_id=6G4JXFD6OOM47PKD3UYXDIYWX6RJ3V35ASKVVF3AFJPZJ7HUAX7A, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.688984956 (554h57m25.560068001s) [noise=true , address=45.91.116.10:4083 , piece_id=CBZGUSSQCQO6MSQTQBX2QCE6YDT5TZYPPL7H5PQDHWTBBDNYAO3A, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.701184987 (554h57m25.572268022s) [noise=true , address=45.91.116.5:4297 , piece_id=NUCPTHPSFAAP57LUOOLKALYTE6D5MEYSQXDMO4UPQX6WLVI3CI5A, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.721573419 (554h57m25.592656455s) [noise=true , address=45.91.116.14:4191 , piece_id=NGSRRP5YODM3SFOYXFOJU472B7QQHZP53CNQBTVBCH3SD4AE63JQ, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.721597786 (554h57m25.592680831s) [seg_pos={0 0}, upload=486540]: Segment reached optimal threshold of 13 pieces. Dec 28 18:26:25.721785623 (554h57m25.592868668s) [noise=true , address=45.91.116.11:4118 , piece_id=LPSKJYE477KZZF4EZ5UA7YWSUTZ3RZEHF54VBR2RD7D3FVXXKBDA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:25.721792247 (554h57m25.592875282s) [noise=true , address=45.91.116.11:4118 , piece_id=LPSKJYE477KZZF4EZ5UA7YWSUTZ3RZEHF54VBR2RD7D3FVXXKBDA, seg_pos={0 0}, upload=486540]: Long tail cancellation due to optimal threshold Dec 28 18:26:25.722050248 (554h57m25.593133283s) [noise=true , address=77.90.31.14:4661 , piece_id=DHN6UF3YP74GS2GC4L6ZMTIKDVDDBMCERIH4EJ46GMMBEMRZ4FPA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:54526->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:54526->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54526->77.90.31.14:4661: use of closed network connection Dec 28 18:26:25.722057131 (554h57m25.593140166s) [noise=true , address=77.90.31.14:4661 , piece_id=DHN6UF3YP74GS2GC4L6ZMTIKDVDDBMCERIH4EJ46GMMBEMRZ4FPA, seg_pos={0 0}, upload=486540]: Long tail cancellation due to optimal threshold Dec 28 18:26:25.722094562 (554h57m25.593177597s) [noise=true , address=45.91.116.6:4975 , piece_id=REHMBIHT2FLO223AJ6ILW27FVGE3U3O3TVIZPM7I65SHWMIPCP7Q, seg_pos={0 0}, upload=486540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:25.722124549 (554h57m25.593207594s) [noise=true , address=45.91.116.6:4975 , piece_id=REHMBIHT2FLO223AJ6ILW27FVGE3U3O3TVIZPM7I65SHWMIPCP7Q, seg_pos={0 0}, upload=486540]: Long tail cancellation due to optimal threshold Dec 28 18:26:25.722155558 (554h57m25.593238603s) [seg_pos={0 0}, upload=486540]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:26:25.722315684 (554h57m25.593398719s) [upload=486540]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:25.722324580 (554h57m25.593407615s) [upload=486540]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:26.014139166 (554h57m25.885222201s) [upload=486540]: Done uploading segments... Dec 28 18:26:26.014159143 (554h57m25.885242188s) [upload=486540]: Upload finished. err: <nil> Duration: 5.8371454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [........X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X.] duration: 5.981627959s trace: https://jaeger.dev.storj.io/trace/3e86b8dbc57740e4 FAIL (2025-12-28 18:27:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: CBZGUSSQCQO6MSQTQBX2QCE6YDT5TZYPPL7H5PQDHWTBBDNYAO3A) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: NGSRRP5YODM3SFOYXFOJU472B7QQHZP53CNQBTVBCH3SD4AE63JQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NUCPTHPSFAAP57LUOOLKALYTE6D5MEYSQXDMO4UPQX6WLVI3CI5A) context deadline exceeded ======LOGS====== 8 18:26:23.223132473 (554h57m23.094215518s) [upload=486540]: Starting upload Dec 28 18:26:23.223206554 (554h57m23.094289619s) [upload=486540]: Done starting upload Dec 28 18:26:23.223392588 (554h57m23.094475633s) [upload=486540]: Uploading segments... Dec 28 18:26:23.223430210 (554h57m23.094513245s) [upload=486540]: Got next segment. Inline: false Dec 28 18:26:23.223458774 (554h57m23.094541819s) [upload=486540]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:26:23.223463824 (554h57m23.094546859s) [upload=486540]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:26:23.466996332 (554h57m23.338079367s) [seg_pos={0 0}, upload=486540]: Begin upload segment... Dec 28 18:26:23.467052268 (554h57m23.338135303s) [seg_pos={0 0}, upload=486540]: Done begin upload segment. Dec 28 18:26:23.467128193 (554h57m23.338211229s) [noise=true , address=145.79.227.50:4373 , piece_id=SBKGFELBFZJUUUP22OM4GLNV5FXPTO4GKWTFKE7NEVQXDIA44JSA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.467137471 (554h57m23.338220516s) [noise=true , address=145.79.227.32:4834 , piece_id=7WP7UODNJ6DPVN6VLHF5PGATODA276DUFLNUJKOD7JY5VGCMOTXA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.467766068 (554h57m23.338849103s) [noise=true , address=145.79.227.9:4366 , piece_id=J5WM6C3LUXXNTNX5APTY3GEONXUMZ5ZPOX3YZLPWCHLRJNPJSZ6Q, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.468245401 (554h57m23.339328446s) [noise=true , address=45.91.116.6:4975 , piece_id=REHMBIHT2FLO223AJ6ILW27FVGE3U3O3TVIZPM7I65SHWMIPCP7Q, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.469190711 (554h57m23.340273766s) [noise=true , address=145.79.227.8:4329 , piece_id=GJ74JSEIN5VQZLON3BBHJF3VJZ7JT23OGDPPGORMIT7ZKTSMIVLQ, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.469979403 (554h57m23.341062448s) [noise=true , address=45.91.116.11:4118 , piece_id=LPSKJYE477KZZF4EZ5UA7YWSUTZ3RZEHF54VBR2RD7D3FVXXKBDA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.471440094 (554h57m23.342523139s) [noise=true , address=145.79.227.21:4473 , piece_id=U7VUTPIX6VEM3IFCEHSIRXVLD2SIDHR4VWSZG2IKWUFBXY6J4FCA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.471872908 (554h57m23.342955943s) [noise=true , address=77.90.31.14:4661 , piece_id=DHN6UF3YP74GS2GC4L6ZMTIKDVDDBMCERIH4EJ46GMMBEMRZ4FPA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.472871359 (554h57m23.343954405s) [noise=true , address=45.91.116.5:4297 , piece_id=NUCPTHPSFAAP57LUOOLKALYTE6D5MEYSQXDMO4UPQX6WLVI3CI5A, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.474102874 (554h57m23.345185919s) [noise=true , address=145.79.227.16:4911 , piece_id=6G4JXFD6OOM47PKD3UYXDIYWX6RJ3V35ASKVVF3AFJPZJ7HUAX7A, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.477653535 (554h57m23.34873657s) [noise=true , address=45.91.116.10:4083 , piece_id=CBZGUSSQCQO6MSQTQBX2QCE6YDT5TZYPPL7H5PQDHWTBBDNYAO3A, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.484617305 (554h57m23.35570035s) [noise=true , address=145.79.227.11:4731 , piece_id=N6OFZ3RG4O64GUH75ANLC42UP5XYDDR7SPC6DK7AGSOIPMIR3JGQ, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.486794101 (554h57m23.357877136s) [noise=true , address=145.79.227.42:4121 , piece_id=IKX3O3K2EQSOUEKEDYI4UDOMFG3LPNHC7BNFPDFD4CEUSD6Z4BIA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.488215697 (554h57m23.359298742s) [noise=true , address=145.79.227.3:4150 , piece_id=5J7TDN4L3RGQLKI27FOD4WKNPVOXZFYZCY4APIBXFXXXKYRXQ4YA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.489318788 (554h57m23.360401823s) [noise=true , address=45.91.116.14:4191 , piece_id=NGSRRP5YODM3SFOYXFOJU472B7QQHZP53CNQBTVBCH3SD4AE63JQ, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.490544642 (554h57m23.361627677s) [noise=true , address=145.79.227.19:4077 , piece_id=O3LJG77WZHLCAFQR73KAKV5YCIQKKEMHDB5SYNZUQW4PDTUOXUVA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:25.196094737 (554h57m25.067177772s) [upload=486540]: Next returned nil segment Dec 28 18:26:25.196119805 (554h57m25.06720284s) [upload=486540]: Waiting for error group managing segments... Dec 28 18:26:25.484279761 (554h57m25.355362796s) [noise=true , address=145.79.227.21:4473 , piece_id=U7VUTPIX6VEM3IFCEHSIRXVLD2SIDHR4VWSZG2IKWUFBXY6J4FCA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.486082404 (554h57m25.357165439s) [noise=true , address=145.79.227.9:4366 , piece_id=J5WM6C3LUXXNTNX5APTY3GEONXUMZ5ZPOX3YZLPWCHLRJNPJSZ6Q, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.488101639 (554h57m25.359184684s) [noise=true , address=145.79.227.8:4329 , piece_id=GJ74JSEIN5VQZLON3BBHJF3VJZ7JT23OGDPPGORMIT7ZKTSMIVLQ, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.488490249 (554h57m25.359573294s) [noise=true , address=145.79.227.32:4834 , piece_id=7WP7UODNJ6DPVN6VLHF5PGATODA276DUFLNUJKOD7JY5VGCMOTXA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.493244001 (554h57m25.364327036s) [noise=true , address=145.79.227.19:4077 , piece_id=O3LJG77WZHLCAFQR73KAKV5YCIQKKEMHDB5SYNZUQW4PDTUOXUVA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.495432689 (554h57m25.366515734s) [noise=true , address=145.79.227.3:4150 , piece_id=5J7TDN4L3RGQLKI27FOD4WKNPVOXZFYZCY4APIBXFXXXKYRXQ4YA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.503725950 (554h57m25.374808995s) [noise=true , address=145.79.227.50:4373 , piece_id=SBKGFELBFZJUUUP22OM4GLNV5FXPTO4GKWTFKE7NEVQXDIA44JSA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.510661887 (554h57m25.381744932s) [noise=true , address=145.79.227.11:4731 , piece_id=N6OFZ3RG4O64GUH75ANLC42UP5XYDDR7SPC6DK7AGSOIPMIR3JGQ, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.517550905 (554h57m25.38863395s) [noise=true , address=145.79.227.42:4121 , piece_id=IKX3O3K2EQSOUEKEDYI4UDOMFG3LPNHC7BNFPDFD4CEUSD6Z4BIA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.545790037 (554h57m25.416873072s) [noise=true , address=145.79.227.16:4911 , piece_id=6G4JXFD6OOM47PKD3UYXDIYWX6RJ3V35ASKVVF3AFJPZJ7HUAX7A, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.688984956 (554h57m25.560068001s) [noise=true , address=45.91.116.10:4083 , piece_id=CBZGUSSQCQO6MSQTQBX2QCE6YDT5TZYPPL7H5PQDHWTBBDNYAO3A, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.701184987 (554h57m25.572268022s) [noise=true , address=45.91.116.5:4297 , piece_id=NUCPTHPSFAAP57LUOOLKALYTE6D5MEYSQXDMO4UPQX6WLVI3CI5A, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.721573419 (554h57m25.592656455s) [noise=true , address=45.91.116.14:4191 , piece_id=NGSRRP5YODM3SFOYXFOJU472B7QQHZP53CNQBTVBCH3SD4AE63JQ, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.721597786 (554h57m25.592680831s) [seg_pos={0 0}, upload=486540]: Segment reached optimal threshold of 13 pieces. Dec 28 18:26:25.721785623 (554h57m25.592868668s) [noise=true , address=45.91.116.11:4118 , piece_id=LPSKJYE477KZZF4EZ5UA7YWSUTZ3RZEHF54VBR2RD7D3FVXXKBDA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:25.721792247 (554h57m25.592875282s) [noise=true , address=45.91.116.11:4118 , piece_id=LPSKJYE477KZZF4EZ5UA7YWSUTZ3RZEHF54VBR2RD7D3FVXXKBDA, seg_pos={0 0}, upload=486540]: Long tail cancellation due to optimal threshold Dec 28 18:26:25.722050248 (554h57m25.593133283s) [noise=true , address=77.90.31.14:4661 , piece_id=DHN6UF3YP74GS2GC4L6ZMTIKDVDDBMCERIH4EJ46GMMBEMRZ4FPA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:54526->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:54526->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54526->77.90.31.14:4661: use of closed network connection Dec 28 18:26:25.722057131 (554h57m25.593140166s) [noise=true , address=77.90.31.14:4661 , piece_id=DHN6UF3YP74GS2GC4L6ZMTIKDVDDBMCERIH4EJ46GMMBEMRZ4FPA, seg_pos={0 0}, upload=486540]: Long tail cancellation due to optimal threshold Dec 28 18:26:25.722094562 (554h57m25.593177597s) [noise=true , address=45.91.116.6:4975 , piece_id=REHMBIHT2FLO223AJ6ILW27FVGE3U3O3TVIZPM7I65SHWMIPCP7Q, seg_pos={0 0}, upload=486540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:25.722124549 (554h57m25.593207594s) [noise=true , address=45.91.116.6:4975 , piece_id=REHMBIHT2FLO223AJ6ILW27FVGE3U3O3TVIZPM7I65SHWMIPCP7Q, seg_pos={0 0}, upload=486540]: Long tail cancellation due to optimal threshold Dec 28 18:26:25.722155558 (554h57m25.593238603s) [seg_pos={0 0}, upload=486540]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:26:25.722315684 (554h57m25.593398719s) [upload=486540]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:25.722324580 (554h57m25.593407615s) [upload=486540]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:26.014139166 (554h57m25.885222201s) [upload=486540]: Done uploading segments... Dec 28 18:26:26.014159143 (554h57m25.885242188s) [upload=486540]: Upload finished. err: <nil> Duration: 5.981650511s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 3.374417327s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:27:09 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.37445003s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X........X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..X] duration: 30.002222529s trace: https://jaeger.dev.storj.io/trace/3e86b8dbc57740e4 FAIL (2025-12-28 18:26:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: CBZGUSSQCQO6MSQTQBX2QCE6YDT5TZYPPL7H5PQDHWTBBDNYAO3A) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: NGSRRP5YODM3SFOYXFOJU472B7QQHZP53CNQBTVBCH3SD4AE63JQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: NUCPTHPSFAAP57LUOOLKALYTE6D5MEYSQXDMO4UPQX6WLVI3CI5A) context deadline exceeded ======LOGS====== 8 18:26:23.223132473 (554h57m23.094215518s) [upload=486540]: Starting upload Dec 28 18:26:23.223206554 (554h57m23.094289619s) [upload=486540]: Done starting upload Dec 28 18:26:23.223392588 (554h57m23.094475633s) [upload=486540]: Uploading segments... Dec 28 18:26:23.223430210 (554h57m23.094513245s) [upload=486540]: Got next segment. Inline: false Dec 28 18:26:23.223458774 (554h57m23.094541819s) [upload=486540]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:26:23.223463824 (554h57m23.094546859s) [upload=486540]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:26:23.466996332 (554h57m23.338079367s) [seg_pos={0 0}, upload=486540]: Begin upload segment... Dec 28 18:26:23.467052268 (554h57m23.338135303s) [seg_pos={0 0}, upload=486540]: Done begin upload segment. Dec 28 18:26:23.467128193 (554h57m23.338211229s) [noise=true , address=145.79.227.50:4373 , piece_id=SBKGFELBFZJUUUP22OM4GLNV5FXPTO4GKWTFKE7NEVQXDIA44JSA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.467137471 (554h57m23.338220516s) [noise=true , address=145.79.227.32:4834 , piece_id=7WP7UODNJ6DPVN6VLHF5PGATODA276DUFLNUJKOD7JY5VGCMOTXA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.467766068 (554h57m23.338849103s) [noise=true , address=145.79.227.9:4366 , piece_id=J5WM6C3LUXXNTNX5APTY3GEONXUMZ5ZPOX3YZLPWCHLRJNPJSZ6Q, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.468245401 (554h57m23.339328446s) [noise=true , address=45.91.116.6:4975 , piece_id=REHMBIHT2FLO223AJ6ILW27FVGE3U3O3TVIZPM7I65SHWMIPCP7Q, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.469190711 (554h57m23.340273766s) [noise=true , address=145.79.227.8:4329 , piece_id=GJ74JSEIN5VQZLON3BBHJF3VJZ7JT23OGDPPGORMIT7ZKTSMIVLQ, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.469979403 (554h57m23.341062448s) [noise=true , address=45.91.116.11:4118 , piece_id=LPSKJYE477KZZF4EZ5UA7YWSUTZ3RZEHF54VBR2RD7D3FVXXKBDA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.471440094 (554h57m23.342523139s) [noise=true , address=145.79.227.21:4473 , piece_id=U7VUTPIX6VEM3IFCEHSIRXVLD2SIDHR4VWSZG2IKWUFBXY6J4FCA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.471872908 (554h57m23.342955943s) [noise=true , address=77.90.31.14:4661 , piece_id=DHN6UF3YP74GS2GC4L6ZMTIKDVDDBMCERIH4EJ46GMMBEMRZ4FPA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.472871359 (554h57m23.343954405s) [noise=true , address=45.91.116.5:4297 , piece_id=NUCPTHPSFAAP57LUOOLKALYTE6D5MEYSQXDMO4UPQX6WLVI3CI5A, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.474102874 (554h57m23.345185919s) [noise=true , address=145.79.227.16:4911 , piece_id=6G4JXFD6OOM47PKD3UYXDIYWX6RJ3V35ASKVVF3AFJPZJ7HUAX7A, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.477653535 (554h57m23.34873657s) [noise=true , address=45.91.116.10:4083 , piece_id=CBZGUSSQCQO6MSQTQBX2QCE6YDT5TZYPPL7H5PQDHWTBBDNYAO3A, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.484617305 (554h57m23.35570035s) [noise=true , address=145.79.227.11:4731 , piece_id=N6OFZ3RG4O64GUH75ANLC42UP5XYDDR7SPC6DK7AGSOIPMIR3JGQ, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.486794101 (554h57m23.357877136s) [noise=true , address=145.79.227.42:4121 , piece_id=IKX3O3K2EQSOUEKEDYI4UDOMFG3LPNHC7BNFPDFD4CEUSD6Z4BIA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.488215697 (554h57m23.359298742s) [noise=true , address=145.79.227.3:4150 , piece_id=5J7TDN4L3RGQLKI27FOD4WKNPVOXZFYZCY4APIBXFXXXKYRXQ4YA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.489318788 (554h57m23.360401823s) [noise=true , address=45.91.116.14:4191 , piece_id=NGSRRP5YODM3SFOYXFOJU472B7QQHZP53CNQBTVBCH3SD4AE63JQ, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:23.490544642 (554h57m23.361627677s) [noise=true , address=145.79.227.19:4077 , piece_id=O3LJG77WZHLCAFQR73KAKV5YCIQKKEMHDB5SYNZUQW4PDTUOXUVA, seg_pos={0 0}, upload=486540]: Uploading piece... Dec 28 18:26:25.196094737 (554h57m25.067177772s) [upload=486540]: Next returned nil segment Dec 28 18:26:25.196119805 (554h57m25.06720284s) [upload=486540]: Waiting for error group managing segments... Dec 28 18:26:25.484279761 (554h57m25.355362796s) [noise=true , address=145.79.227.21:4473 , piece_id=U7VUTPIX6VEM3IFCEHSIRXVLD2SIDHR4VWSZG2IKWUFBXY6J4FCA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.486082404 (554h57m25.357165439s) [noise=true , address=145.79.227.9:4366 , piece_id=J5WM6C3LUXXNTNX5APTY3GEONXUMZ5ZPOX3YZLPWCHLRJNPJSZ6Q, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.488101639 (554h57m25.359184684s) [noise=true , address=145.79.227.8:4329 , piece_id=GJ74JSEIN5VQZLON3BBHJF3VJZ7JT23OGDPPGORMIT7ZKTSMIVLQ, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.488490249 (554h57m25.359573294s) [noise=true , address=145.79.227.32:4834 , piece_id=7WP7UODNJ6DPVN6VLHF5PGATODA276DUFLNUJKOD7JY5VGCMOTXA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.493244001 (554h57m25.364327036s) [noise=true , address=145.79.227.19:4077 , piece_id=O3LJG77WZHLCAFQR73KAKV5YCIQKKEMHDB5SYNZUQW4PDTUOXUVA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.495432689 (554h57m25.366515734s) [noise=true , address=145.79.227.3:4150 , piece_id=5J7TDN4L3RGQLKI27FOD4WKNPVOXZFYZCY4APIBXFXXXKYRXQ4YA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.503725950 (554h57m25.374808995s) [noise=true , address=145.79.227.50:4373 , piece_id=SBKGFELBFZJUUUP22OM4GLNV5FXPTO4GKWTFKE7NEVQXDIA44JSA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.510661887 (554h57m25.381744932s) [noise=true , address=145.79.227.11:4731 , piece_id=N6OFZ3RG4O64GUH75ANLC42UP5XYDDR7SPC6DK7AGSOIPMIR3JGQ, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.517550905 (554h57m25.38863395s) [noise=true , address=145.79.227.42:4121 , piece_id=IKX3O3K2EQSOUEKEDYI4UDOMFG3LPNHC7BNFPDFD4CEUSD6Z4BIA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.545790037 (554h57m25.416873072s) [noise=true , address=145.79.227.16:4911 , piece_id=6G4JXFD6OOM47PKD3UYXDIYWX6RJ3V35ASKVVF3AFJPZJ7HUAX7A, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.688984956 (554h57m25.560068001s) [noise=true , address=45.91.116.10:4083 , piece_id=CBZGUSSQCQO6MSQTQBX2QCE6YDT5TZYPPL7H5PQDHWTBBDNYAO3A, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.701184987 (554h57m25.572268022s) [noise=true , address=45.91.116.5:4297 , piece_id=NUCPTHPSFAAP57LUOOLKALYTE6D5MEYSQXDMO4UPQX6WLVI3CI5A, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.721573419 (554h57m25.592656455s) [noise=true , address=45.91.116.14:4191 , piece_id=NGSRRP5YODM3SFOYXFOJU472B7QQHZP53CNQBTVBCH3SD4AE63JQ, seg_pos={0 0}, upload=486540]: Done uploading piece. err: <nil> Dec 28 18:26:25.721597786 (554h57m25.592680831s) [seg_pos={0 0}, upload=486540]: Segment reached optimal threshold of 13 pieces. Dec 28 18:26:25.721785623 (554h57m25.592868668s) [noise=true , address=45.91.116.11:4118 , piece_id=LPSKJYE477KZZF4EZ5UA7YWSUTZ3RZEHF54VBR2RD7D3FVXXKBDA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:25.721792247 (554h57m25.592875282s) [noise=true , address=45.91.116.11:4118 , piece_id=LPSKJYE477KZZF4EZ5UA7YWSUTZ3RZEHF54VBR2RD7D3FVXXKBDA, seg_pos={0 0}, upload=486540]: Long tail cancellation due to optimal threshold Dec 28 18:26:25.722050248 (554h57m25.593133283s) [noise=true , address=77.90.31.14:4661 , piece_id=DHN6UF3YP74GS2GC4L6ZMTIKDVDDBMCERIH4EJ46GMMBEMRZ4FPA, seg_pos={0 0}, upload=486540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:54526->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:54526->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54526->77.90.31.14:4661: use of closed network connection Dec 28 18:26:25.722057131 (554h57m25.593140166s) [noise=true , address=77.90.31.14:4661 , piece_id=DHN6UF3YP74GS2GC4L6ZMTIKDVDDBMCERIH4EJ46GMMBEMRZ4FPA, seg_pos={0 0}, upload=486540]: Long tail cancellation due to optimal threshold Dec 28 18:26:25.722094562 (554h57m25.593177597s) [noise=true , address=45.91.116.6:4975 , piece_id=REHMBIHT2FLO223AJ6ILW27FVGE3U3O3TVIZPM7I65SHWMIPCP7Q, seg_pos={0 0}, upload=486540]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 18:26:25.722124549 (554h57m25.593207594s) [noise=true , address=45.91.116.6:4975 , piece_id=REHMBIHT2FLO223AJ6ILW27FVGE3U3O3TVIZPM7I65SHWMIPCP7Q, seg_pos={0 0}, upload=486540]: Long tail cancellation due to optimal threshold Dec 28 18:26:25.722155558 (554h57m25.593238603s) [seg_pos={0 0}, upload=486540]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:26:25.722315684 (554h57m25.593398719s) [upload=486540]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:26:25.722324580 (554h57m25.593407615s) [upload=486540]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:26:26.014139166 (554h57m25.885222201s) [upload=486540]: Done uploading segments... Dec 28 18:26:26.014159143 (554h57m25.885242188s) [upload=486540]: Upload finished. err: <nil> Duration: 30.002242867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X........X.............X.X........................] duration: 48.74356431s trace: https://jaeger.dev.storj.io/trace/5eda81bca04798b7 FAIL (2025-12-28 18:26:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794293523 (554h35m21.665376568s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794729272 (554h35m21.665812307s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795130186 (554h35m21.666213231s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795534857 (554h35m21.666617902s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795928286 (554h35m21.667011331s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796356250 (554h35m21.667439285s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796770529 (554h35m21.667853574s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797234142 (554h35m21.668317187s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797701582 (554h35m21.668784697s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798145758 (554h35m21.669228793s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798717497 (554h35m21.669800532s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799245331 (554h35m21.670328377s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799828242 (554h35m21.670911287s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.800407144 (554h35m21.671490189s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.801046071 (554h35m21.672129116s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:23.074697857 (554h35m22.945780892s) [upload=486222]: Next returned nil segment Dec 28 18:04:23.074711302 (554h35m22.945794337s) [upload=486222]: Waiting for error group managing segments... Dec 28 18:04:23.417354815 (554h35m23.28843785s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.486601663 (554h35m23.357684708s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.668116249 (554h35m23.539199284s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.843876887 (554h35m23.714959922s) [noise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.848651940 (554h35m23.719734985s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.875480346 (554h35m23.746563381s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.029712621 (554h35m23.900795666s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.428381026 (554h35m24.299464061s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.550713979 (554h35m24.421797024s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.561873249 (554h35m24.432956284s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.583710861 (554h35m24.454793896s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.596115952 (554h35m24.467198997s) [noise=true , address=145.79.227.29:4959 , piece_id=F4UQHHNYPJVFEF336W3V5HIEPBKRQWM4DKOJAEXVHGULYWHLK75A, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.630336734 (554h35m24.501419779s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681392994 (554h35m24.552476029s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681422491 (554h35m24.552505556s) [seg_pos={0 0}, upload=486222]: Segment reached optimal threshold of 14 pieces. Dec 28 18:04:24.681622883 (554h35m24.552705918s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection Dec 28 18:04:24.681629966 (554h35m24.552713011s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681732601 (554h35m24.552815646s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681742510 (554h35m24.552825555s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681912925 (554h35m24.55299596s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681919507 (554h35m24.553002542s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682191224 (554h35m24.55327427s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.682198829 (554h35m24.553281864s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682218246 (554h35m24.553301281s) [seg_pos={0 0}, upload=486222]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:04:24.682292608 (554h35m24.553375653s) [upload=486222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:04:24.682297497 (554h35m24.553380532s) [upload=486222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:04:24.980627610 (554h35m24.851710655s) [upload=486222]: Done uploading segments... Dec 28 18:04:24.980654070 (554h35m24.851737105s) [upload=486222]: Upload finished. err: <nil> Duration: 48.743594159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 3.128487574s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:26:03 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.128532861s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX........X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX..] duration: 5.847403903s trace: https://jaeger.dev.storj.io/trace/2b0f83b9ad19f1e7 FAIL (2025-12-28 18:25:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 28 18:23:34.051987230 (554h54m33.923070265s) [upload=486493]: Got next segment. Inline: false Dec 28 18:23:34.052064768 (554h54m33.923147803s) [upload=486493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:23:34.052071160 (554h54m33.923154205s) [upload=486493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:23:34.303824312 (554h54m34.174907357s) [seg_pos={0 0}, upload=486493]: Begin upload segment... Dec 28 18:23:34.303871561 (554h54m34.174954596s) [seg_pos={0 0}, upload=486493]: Done begin upload segment. Dec 28 18:23:34.303900967 (554h54m34.174984002s) [noise=true , address=45.91.116.8:4046 , piece_id=HURGUSAGZEJX2GEUBLVWZ4B2FGM4TRF6CB7VTCOYAXTSSZYUZJNQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.303936064 (554h54m34.175019099s) [noise=true , address=145.79.227.45:4230 , piece_id=HW4GVTBOOLWNWFB2JTYDTVHVDZRPUYXTDYHYFVCTJFJVGX5IZOFQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.304448350 (554h54m34.175531405s) [noise=true , address=145.79.227.51:4411 , piece_id=JDGKATQEEK7AXN4NMKBFT3EV2CBUKXXOYXZFV6LU3SI3XZADYYPA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.304900461 (554h54m34.175983496s) [noise=true , address=145.79.227.11:4732 , piece_id=ED2TFFO4XNROP3CVF7TDS7WLQEZVLFK7VGDLPST73ZRFIUEXFMZA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.305303227 (554h54m34.176386262s) [noise=true , address=145.79.227.12:4765 , piece_id=HK7SHSNCV7JB34E3HPHSHMN7UPKUTAZGESJYAH2TMXQPI3QUH57A, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.305654015 (554h54m34.17673706s) [noise=true , address=145.79.227.52:4445 , piece_id=4KD2OETYNY2QF53CWU3P5SVVXX4WADMR7FNR65S5O6RVW5T2TMLA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.306010905 (554h54m34.177093951s) [noise=true , address=45.91.116.6:4975 , piece_id=SZZNUDKG3XP6XED2ZI2TZ2ESVNZ5K3DVEEEEVPFSOSZG5DCIP4ZQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.306482212 (554h54m34.177565257s) [noise=true , address=145.79.227.36:4519 , piece_id=OBQSHCMXZXDYWO5ZQ22F7SZFBFZ55DURP6TCTSY533YV7PD2G66A, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.306856926 (554h54m34.177939961s) [noise=true , address=45.91.116.12:4228 , piece_id=SMOYJWFDTCBDAMBZVNBO5RLV7UPOGYYVF7QXOXKZZGITZOXIIE6A, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.307229055 (554h54m34.17831209s) [noise=true , address=145.79.227.4:4185 , piece_id=QXZHEH2MBJ5V7UGV3HOIUXJCB5QU2Z3YX4UYPQETQLY3BR76WTIA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.307578270 (554h54m34.178661305s) [noise=true , address=145.79.227.26:4653 , piece_id=PGPQ6JCA4M3N6K3MMTFIMMNY2FOKV36PHTWVDNLLDVZP2Z3BPSCQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.307943395 (554h54m34.17902643s) [noise=true , address=77.90.31.19:4841 , piece_id=C4CUPNTQATSUMO26I3I2A4NW65ASY6EJUPR4A767FMJFBBYNLNMQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.308301035 (554h54m34.179384081s) [noise=true , address=145.79.227.8:4329 , piece_id=AL6PEJZQVEB4XIME62LE42OIT7IOMRKCVSSMHQ2RH4EPBRPSBC5Q, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.310065596 (554h54m34.181148651s) [noise=true , address=145.79.227.19:4077 , piece_id=MEDDJAJRK7GFKCRET4JIO4UJI6UW4WPG2PJ5E6CPEDVIFD6DOONA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.310818109 (554h54m34.181901154s) [noise=true , address=45.91.116.4:4939 , piece_id=OKJG2NIUUNIROT7B5FZQCO2CECTCJRGVLPDWWTJ45QD4TKDM66IQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.311369219 (554h54m34.182452264s) [noise=true , address=77.90.31.18:4808 , piece_id=3BIJMDJ5GSBWPFQIVAVO74O7BSJFCMNKEHYAF4HUNSXA726F7VLQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:36.016711256 (554h54m35.887794291s) [upload=486493]: Next returned nil segment Dec 28 18:23:36.016728288 (554h54m35.887811323s) [upload=486493]: Waiting for error group managing segments... Dec 28 18:23:36.299126394 (554h54m36.170209429s) [noise=true , address=145.79.227.51:4411 , piece_id=JDGKATQEEK7AXN4NMKBFT3EV2CBUKXXOYXZFV6LU3SI3XZADYYPA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.310748504 (554h54m36.181831539s) [noise=true , address=145.79.227.45:4230 , piece_id=HW4GVTBOOLWNWFB2JTYDTVHVDZRPUYXTDYHYFVCTJFJVGX5IZOFQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.315988061 (554h54m36.187071106s) [noise=true , address=145.79.227.8:4329 , piece_id=AL6PEJZQVEB4XIME62LE42OIT7IOMRKCVSSMHQ2RH4EPBRPSBC5Q, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.321954332 (554h54m36.193037377s) [noise=true , address=145.79.227.36:4519 , piece_id=OBQSHCMXZXDYWO5ZQ22F7SZFBFZ55DURP6TCTSY533YV7PD2G66A, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.333499867 (554h54m36.204582902s) [noise=true , address=145.79.227.19:4077 , piece_id=MEDDJAJRK7GFKCRET4JIO4UJI6UW4WPG2PJ5E6CPEDVIFD6DOONA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.341732713 (554h54m36.212815749s) [noise=true , address=145.79.227.52:4445 , piece_id=4KD2OETYNY2QF53CWU3P5SVVXX4WADMR7FNR65S5O6RVW5T2TMLA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.343215437 (554h54m36.214298472s) [noise=true , address=145.79.227.11:4732 , piece_id=ED2TFFO4XNROP3CVF7TDS7WLQEZVLFK7VGDLPST73ZRFIUEXFMZA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.491781633 (554h54m36.362864668s) [noise=true , address=145.79.227.12:4765 , piece_id=HK7SHSNCV7JB34E3HPHSHMN7UPKUTAZGESJYAH2TMXQPI3QUH57A, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.517957315 (554h54m36.38904035s) [noise=true , address=45.91.116.12:4228 , piece_id=SMOYJWFDTCBDAMBZVNBO5RLV7UPOGYYVF7QXOXKZZGITZOXIIE6A, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.588144384 (554h54m36.459227429s) [noise=true , address=45.91.116.8:4046 , piece_id=HURGUSAGZEJX2GEUBLVWZ4B2FGM4TRF6CB7VTCOYAXTSSZYUZJNQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.601871713 (554h54m36.472954758s) [noise=true , address=77.90.31.19:4841 , piece_id=C4CUPNTQATSUMO26I3I2A4NW65ASY6EJUPR4A767FMJFBBYNLNMQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.616576253 (554h54m36.487659288s) [noise=true , address=45.91.116.4:4939 , piece_id=OKJG2NIUUNIROT7B5FZQCO2CECTCJRGVLPDWWTJ45QD4TKDM66IQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.937330425 (554h54m36.80841346s) [noise=true , address=45.91.116.6:4975 , piece_id=SZZNUDKG3XP6XED2ZI2TZ2ESVNZ5K3DVEEEEVPFSOSZG5DCIP4ZQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.937352146 (554h54m36.808435192s) [seg_pos={0 0}, upload=486493]: Segment reached optimal threshold of 13 pieces. Dec 28 18:23:36.937561114 (554h54m36.808644149s) [noise=true , address=145.79.227.4:4185 , piece_id=QXZHEH2MBJ5V7UGV3HOIUXJCB5QU2Z3YX4UYPQETQLY3BR76WTIA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:60318->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:60318->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60318->145.79.227.4:4185: use of closed network connection Dec 28 18:23:36.937569139 (554h54m36.808652175s) [noise=true , address=145.79.227.4:4185 , piece_id=QXZHEH2MBJ5V7UGV3HOIUXJCB5QU2Z3YX4UYPQETQLY3BR76WTIA, seg_pos={0 0}, upload=486493]: Long tail cancellation due to optimal threshold Dec 28 18:23:36.937619125 (554h54m36.80870216s) [noise=true , address=77.90.31.18:4808 , piece_id=3BIJMDJ5GSBWPFQIVAVO74O7BSJFCMNKEHYAF4HUNSXA726F7VLQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 18:23:36.937629044 (554h54m36.808712079s) [noise=true , address=77.90.31.18:4808 , piece_id=3BIJMDJ5GSBWPFQIVAVO74O7BSJFCMNKEHYAF4HUNSXA726F7VLQ, seg_pos={0 0}, upload=486493]: Long tail cancellation due to optimal threshold Dec 28 18:23:36.937969122 (554h54m36.809052157s) [noise=true , address=145.79.227.26:4653 , piece_id=PGPQ6JCA4M3N6K3MMTFIMMNY2FOKV36PHTWVDNLLDVZP2Z3BPSCQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:54790->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:54790->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54790->145.79.227.26:4653: use of closed network connection Dec 28 18:23:36.937976586 (554h54m36.809059631s) [noise=true , address=145.79.227.26:4653 , piece_id=PGPQ6JCA4M3N6K3MMTFIMMNY2FOKV36PHTWVDNLLDVZP2Z3BPSCQ, seg_pos={0 0}, upload=486493]: Long tail cancellation due to optimal threshold Dec 28 18:23:36.938087917 (554h54m36.809170952s) [seg_pos={0 0}, upload=486493]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:23:36.938162369 (554h54m36.809245414s) [upload=486493]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:23:36.938167128 (554h54m36.809250163s) [upload=486493]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:23:37.224268686 (554h54m37.095351721s) [upload=486493]: Done uploading segments... Dec 28 18:23:37.224295466 (554h54m37.095378511s) [upload=486493]: Upload finished. err: <nil> Duration: 5.847432007s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 3.326817189s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:25:32 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.326842867s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX........X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX.] duration: 6.133785053s trace: https://jaeger.dev.storj.io/trace/2b0f83b9ad19f1e7 FAIL (2025-12-28 18:24:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 28 18:23:34.051987230 (554h54m33.923070265s) [upload=486493]: Got next segment. Inline: false Dec 28 18:23:34.052064768 (554h54m33.923147803s) [upload=486493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:23:34.052071160 (554h54m33.923154205s) [upload=486493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:23:34.303824312 (554h54m34.174907357s) [seg_pos={0 0}, upload=486493]: Begin upload segment... Dec 28 18:23:34.303871561 (554h54m34.174954596s) [seg_pos={0 0}, upload=486493]: Done begin upload segment. Dec 28 18:23:34.303900967 (554h54m34.174984002s) [noise=true , address=45.91.116.8:4046 , piece_id=HURGUSAGZEJX2GEUBLVWZ4B2FGM4TRF6CB7VTCOYAXTSSZYUZJNQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.303936064 (554h54m34.175019099s) [noise=true , address=145.79.227.45:4230 , piece_id=HW4GVTBOOLWNWFB2JTYDTVHVDZRPUYXTDYHYFVCTJFJVGX5IZOFQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.304448350 (554h54m34.175531405s) [noise=true , address=145.79.227.51:4411 , piece_id=JDGKATQEEK7AXN4NMKBFT3EV2CBUKXXOYXZFV6LU3SI3XZADYYPA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.304900461 (554h54m34.175983496s) [noise=true , address=145.79.227.11:4732 , piece_id=ED2TFFO4XNROP3CVF7TDS7WLQEZVLFK7VGDLPST73ZRFIUEXFMZA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.305303227 (554h54m34.176386262s) [noise=true , address=145.79.227.12:4765 , piece_id=HK7SHSNCV7JB34E3HPHSHMN7UPKUTAZGESJYAH2TMXQPI3QUH57A, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.305654015 (554h54m34.17673706s) [noise=true , address=145.79.227.52:4445 , piece_id=4KD2OETYNY2QF53CWU3P5SVVXX4WADMR7FNR65S5O6RVW5T2TMLA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.306010905 (554h54m34.177093951s) [noise=true , address=45.91.116.6:4975 , piece_id=SZZNUDKG3XP6XED2ZI2TZ2ESVNZ5K3DVEEEEVPFSOSZG5DCIP4ZQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.306482212 (554h54m34.177565257s) [noise=true , address=145.79.227.36:4519 , piece_id=OBQSHCMXZXDYWO5ZQ22F7SZFBFZ55DURP6TCTSY533YV7PD2G66A, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.306856926 (554h54m34.177939961s) [noise=true , address=45.91.116.12:4228 , piece_id=SMOYJWFDTCBDAMBZVNBO5RLV7UPOGYYVF7QXOXKZZGITZOXIIE6A, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.307229055 (554h54m34.17831209s) [noise=true , address=145.79.227.4:4185 , piece_id=QXZHEH2MBJ5V7UGV3HOIUXJCB5QU2Z3YX4UYPQETQLY3BR76WTIA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.307578270 (554h54m34.178661305s) [noise=true , address=145.79.227.26:4653 , piece_id=PGPQ6JCA4M3N6K3MMTFIMMNY2FOKV36PHTWVDNLLDVZP2Z3BPSCQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.307943395 (554h54m34.17902643s) [noise=true , address=77.90.31.19:4841 , piece_id=C4CUPNTQATSUMO26I3I2A4NW65ASY6EJUPR4A767FMJFBBYNLNMQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.308301035 (554h54m34.179384081s) [noise=true , address=145.79.227.8:4329 , piece_id=AL6PEJZQVEB4XIME62LE42OIT7IOMRKCVSSMHQ2RH4EPBRPSBC5Q, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.310065596 (554h54m34.181148651s) [noise=true , address=145.79.227.19:4077 , piece_id=MEDDJAJRK7GFKCRET4JIO4UJI6UW4WPG2PJ5E6CPEDVIFD6DOONA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.310818109 (554h54m34.181901154s) [noise=true , address=45.91.116.4:4939 , piece_id=OKJG2NIUUNIROT7B5FZQCO2CECTCJRGVLPDWWTJ45QD4TKDM66IQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.311369219 (554h54m34.182452264s) [noise=true , address=77.90.31.18:4808 , piece_id=3BIJMDJ5GSBWPFQIVAVO74O7BSJFCMNKEHYAF4HUNSXA726F7VLQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:36.016711256 (554h54m35.887794291s) [upload=486493]: Next returned nil segment Dec 28 18:23:36.016728288 (554h54m35.887811323s) [upload=486493]: Waiting for error group managing segments... Dec 28 18:23:36.299126394 (554h54m36.170209429s) [noise=true , address=145.79.227.51:4411 , piece_id=JDGKATQEEK7AXN4NMKBFT3EV2CBUKXXOYXZFV6LU3SI3XZADYYPA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.310748504 (554h54m36.181831539s) [noise=true , address=145.79.227.45:4230 , piece_id=HW4GVTBOOLWNWFB2JTYDTVHVDZRPUYXTDYHYFVCTJFJVGX5IZOFQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.315988061 (554h54m36.187071106s) [noise=true , address=145.79.227.8:4329 , piece_id=AL6PEJZQVEB4XIME62LE42OIT7IOMRKCVSSMHQ2RH4EPBRPSBC5Q, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.321954332 (554h54m36.193037377s) [noise=true , address=145.79.227.36:4519 , piece_id=OBQSHCMXZXDYWO5ZQ22F7SZFBFZ55DURP6TCTSY533YV7PD2G66A, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.333499867 (554h54m36.204582902s) [noise=true , address=145.79.227.19:4077 , piece_id=MEDDJAJRK7GFKCRET4JIO4UJI6UW4WPG2PJ5E6CPEDVIFD6DOONA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.341732713 (554h54m36.212815749s) [noise=true , address=145.79.227.52:4445 , piece_id=4KD2OETYNY2QF53CWU3P5SVVXX4WADMR7FNR65S5O6RVW5T2TMLA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.343215437 (554h54m36.214298472s) [noise=true , address=145.79.227.11:4732 , piece_id=ED2TFFO4XNROP3CVF7TDS7WLQEZVLFK7VGDLPST73ZRFIUEXFMZA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.491781633 (554h54m36.362864668s) [noise=true , address=145.79.227.12:4765 , piece_id=HK7SHSNCV7JB34E3HPHSHMN7UPKUTAZGESJYAH2TMXQPI3QUH57A, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.517957315 (554h54m36.38904035s) [noise=true , address=45.91.116.12:4228 , piece_id=SMOYJWFDTCBDAMBZVNBO5RLV7UPOGYYVF7QXOXKZZGITZOXIIE6A, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.588144384 (554h54m36.459227429s) [noise=true , address=45.91.116.8:4046 , piece_id=HURGUSAGZEJX2GEUBLVWZ4B2FGM4TRF6CB7VTCOYAXTSSZYUZJNQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.601871713 (554h54m36.472954758s) [noise=true , address=77.90.31.19:4841 , piece_id=C4CUPNTQATSUMO26I3I2A4NW65ASY6EJUPR4A767FMJFBBYNLNMQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.616576253 (554h54m36.487659288s) [noise=true , address=45.91.116.4:4939 , piece_id=OKJG2NIUUNIROT7B5FZQCO2CECTCJRGVLPDWWTJ45QD4TKDM66IQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.937330425 (554h54m36.80841346s) [noise=true , address=45.91.116.6:4975 , piece_id=SZZNUDKG3XP6XED2ZI2TZ2ESVNZ5K3DVEEEEVPFSOSZG5DCIP4ZQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.937352146 (554h54m36.808435192s) [seg_pos={0 0}, upload=486493]: Segment reached optimal threshold of 13 pieces. Dec 28 18:23:36.937561114 (554h54m36.808644149s) [noise=true , address=145.79.227.4:4185 , piece_id=QXZHEH2MBJ5V7UGV3HOIUXJCB5QU2Z3YX4UYPQETQLY3BR76WTIA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:60318->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:60318->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60318->145.79.227.4:4185: use of closed network connection Dec 28 18:23:36.937569139 (554h54m36.808652175s) [noise=true , address=145.79.227.4:4185 , piece_id=QXZHEH2MBJ5V7UGV3HOIUXJCB5QU2Z3YX4UYPQETQLY3BR76WTIA, seg_pos={0 0}, upload=486493]: Long tail cancellation due to optimal threshold Dec 28 18:23:36.937619125 (554h54m36.80870216s) [noise=true , address=77.90.31.18:4808 , piece_id=3BIJMDJ5GSBWPFQIVAVO74O7BSJFCMNKEHYAF4HUNSXA726F7VLQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 18:23:36.937629044 (554h54m36.808712079s) [noise=true , address=77.90.31.18:4808 , piece_id=3BIJMDJ5GSBWPFQIVAVO74O7BSJFCMNKEHYAF4HUNSXA726F7VLQ, seg_pos={0 0}, upload=486493]: Long tail cancellation due to optimal threshold Dec 28 18:23:36.937969122 (554h54m36.809052157s) [noise=true , address=145.79.227.26:4653 , piece_id=PGPQ6JCA4M3N6K3MMTFIMMNY2FOKV36PHTWVDNLLDVZP2Z3BPSCQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:54790->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:54790->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54790->145.79.227.26:4653: use of closed network connection Dec 28 18:23:36.937976586 (554h54m36.809059631s) [noise=true , address=145.79.227.26:4653 , piece_id=PGPQ6JCA4M3N6K3MMTFIMMNY2FOKV36PHTWVDNLLDVZP2Z3BPSCQ, seg_pos={0 0}, upload=486493]: Long tail cancellation due to optimal threshold Dec 28 18:23:36.938087917 (554h54m36.809170952s) [seg_pos={0 0}, upload=486493]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:23:36.938162369 (554h54m36.809245414s) [upload=486493]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:23:36.938167128 (554h54m36.809250163s) [upload=486493]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:23:37.224268686 (554h54m37.095351721s) [upload=486493]: Done uploading segments... Dec 28 18:23:37.224295466 (554h54m37.095378511s) [upload=486493]: Upload finished. err: <nil> Duration: 6.133813057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X........X.............X.X.......................] duration: 21.425776985s trace: https://jaeger.dev.storj.io/trace/5eda81bca04798b7 FAIL (2025-12-28 18:24:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794293523 (554h35m21.665376568s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794729272 (554h35m21.665812307s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795130186 (554h35m21.666213231s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795534857 (554h35m21.666617902s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795928286 (554h35m21.667011331s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796356250 (554h35m21.667439285s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796770529 (554h35m21.667853574s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797234142 (554h35m21.668317187s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797701582 (554h35m21.668784697s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798145758 (554h35m21.669228793s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798717497 (554h35m21.669800532s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799245331 (554h35m21.670328377s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799828242 (554h35m21.670911287s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.800407144 (554h35m21.671490189s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.801046071 (554h35m21.672129116s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:23.074697857 (554h35m22.945780892s) [upload=486222]: Next returned nil segment Dec 28 18:04:23.074711302 (554h35m22.945794337s) [upload=486222]: Waiting for error group managing segments... Dec 28 18:04:23.417354815 (554h35m23.28843785s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.486601663 (554h35m23.357684708s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.668116249 (554h35m23.539199284s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.843876887 (554h35m23.714959922s) [noise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.848651940 (554h35m23.719734985s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.875480346 (554h35m23.746563381s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.029712621 (554h35m23.900795666s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.428381026 (554h35m24.299464061s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.550713979 (554h35m24.421797024s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.561873249 (554h35m24.432956284s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.583710861 (554h35m24.454793896s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.596115952 (554h35m24.467198997s) [noise=true , address=145.79.227.29:4959 , piece_id=F4UQHHNYPJVFEF336W3V5HIEPBKRQWM4DKOJAEXVHGULYWHLK75A, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.630336734 (554h35m24.501419779s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681392994 (554h35m24.552476029s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681422491 (554h35m24.552505556s) [seg_pos={0 0}, upload=486222]: Segment reached optimal threshold of 14 pieces. Dec 28 18:04:24.681622883 (554h35m24.552705918s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection Dec 28 18:04:24.681629966 (554h35m24.552713011s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681732601 (554h35m24.552815646s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681742510 (554h35m24.552825555s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681912925 (554h35m24.55299596s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681919507 (554h35m24.553002542s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682191224 (554h35m24.55327427s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.682198829 (554h35m24.553281864s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682218246 (554h35m24.553301281s) [seg_pos={0 0}, upload=486222]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:04:24.682292608 (554h35m24.553375653s) [upload=486222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:04:24.682297497 (554h35m24.553380532s) [upload=486222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:04:24.980627610 (554h35m24.851710655s) [upload=486222]: Done uploading segments... Dec 28 18:04:24.980654070 (554h35m24.851737105s) [upload=486222]: Upload finished. err: <nil> Duration: 21.425806972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 3.37472889s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:24:23 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.374759148s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XX........X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.XX] duration: 30.003076604s trace: https://jaeger.dev.storj.io/trace/2b0f83b9ad19f1e7 FAIL (2025-12-28 18:24:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 28 18:23:34.051987230 (554h54m33.923070265s) [upload=486493]: Got next segment. Inline: false Dec 28 18:23:34.052064768 (554h54m33.923147803s) [upload=486493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:23:34.052071160 (554h54m33.923154205s) [upload=486493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:23:34.303824312 (554h54m34.174907357s) [seg_pos={0 0}, upload=486493]: Begin upload segment... Dec 28 18:23:34.303871561 (554h54m34.174954596s) [seg_pos={0 0}, upload=486493]: Done begin upload segment. Dec 28 18:23:34.303900967 (554h54m34.174984002s) [noise=true , address=45.91.116.8:4046 , piece_id=HURGUSAGZEJX2GEUBLVWZ4B2FGM4TRF6CB7VTCOYAXTSSZYUZJNQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.303936064 (554h54m34.175019099s) [noise=true , address=145.79.227.45:4230 , piece_id=HW4GVTBOOLWNWFB2JTYDTVHVDZRPUYXTDYHYFVCTJFJVGX5IZOFQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.304448350 (554h54m34.175531405s) [noise=true , address=145.79.227.51:4411 , piece_id=JDGKATQEEK7AXN4NMKBFT3EV2CBUKXXOYXZFV6LU3SI3XZADYYPA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.304900461 (554h54m34.175983496s) [noise=true , address=145.79.227.11:4732 , piece_id=ED2TFFO4XNROP3CVF7TDS7WLQEZVLFK7VGDLPST73ZRFIUEXFMZA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.305303227 (554h54m34.176386262s) [noise=true , address=145.79.227.12:4765 , piece_id=HK7SHSNCV7JB34E3HPHSHMN7UPKUTAZGESJYAH2TMXQPI3QUH57A, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.305654015 (554h54m34.17673706s) [noise=true , address=145.79.227.52:4445 , piece_id=4KD2OETYNY2QF53CWU3P5SVVXX4WADMR7FNR65S5O6RVW5T2TMLA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.306010905 (554h54m34.177093951s) [noise=true , address=45.91.116.6:4975 , piece_id=SZZNUDKG3XP6XED2ZI2TZ2ESVNZ5K3DVEEEEVPFSOSZG5DCIP4ZQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.306482212 (554h54m34.177565257s) [noise=true , address=145.79.227.36:4519 , piece_id=OBQSHCMXZXDYWO5ZQ22F7SZFBFZ55DURP6TCTSY533YV7PD2G66A, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.306856926 (554h54m34.177939961s) [noise=true , address=45.91.116.12:4228 , piece_id=SMOYJWFDTCBDAMBZVNBO5RLV7UPOGYYVF7QXOXKZZGITZOXIIE6A, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.307229055 (554h54m34.17831209s) [noise=true , address=145.79.227.4:4185 , piece_id=QXZHEH2MBJ5V7UGV3HOIUXJCB5QU2Z3YX4UYPQETQLY3BR76WTIA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.307578270 (554h54m34.178661305s) [noise=true , address=145.79.227.26:4653 , piece_id=PGPQ6JCA4M3N6K3MMTFIMMNY2FOKV36PHTWVDNLLDVZP2Z3BPSCQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.307943395 (554h54m34.17902643s) [noise=true , address=77.90.31.19:4841 , piece_id=C4CUPNTQATSUMO26I3I2A4NW65ASY6EJUPR4A767FMJFBBYNLNMQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.308301035 (554h54m34.179384081s) [noise=true , address=145.79.227.8:4329 , piece_id=AL6PEJZQVEB4XIME62LE42OIT7IOMRKCVSSMHQ2RH4EPBRPSBC5Q, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.310065596 (554h54m34.181148651s) [noise=true , address=145.79.227.19:4077 , piece_id=MEDDJAJRK7GFKCRET4JIO4UJI6UW4WPG2PJ5E6CPEDVIFD6DOONA, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.310818109 (554h54m34.181901154s) [noise=true , address=45.91.116.4:4939 , piece_id=OKJG2NIUUNIROT7B5FZQCO2CECTCJRGVLPDWWTJ45QD4TKDM66IQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:34.311369219 (554h54m34.182452264s) [noise=true , address=77.90.31.18:4808 , piece_id=3BIJMDJ5GSBWPFQIVAVO74O7BSJFCMNKEHYAF4HUNSXA726F7VLQ, seg_pos={0 0}, upload=486493]: Uploading piece... Dec 28 18:23:36.016711256 (554h54m35.887794291s) [upload=486493]: Next returned nil segment Dec 28 18:23:36.016728288 (554h54m35.887811323s) [upload=486493]: Waiting for error group managing segments... Dec 28 18:23:36.299126394 (554h54m36.170209429s) [noise=true , address=145.79.227.51:4411 , piece_id=JDGKATQEEK7AXN4NMKBFT3EV2CBUKXXOYXZFV6LU3SI3XZADYYPA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.310748504 (554h54m36.181831539s) [noise=true , address=145.79.227.45:4230 , piece_id=HW4GVTBOOLWNWFB2JTYDTVHVDZRPUYXTDYHYFVCTJFJVGX5IZOFQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.315988061 (554h54m36.187071106s) [noise=true , address=145.79.227.8:4329 , piece_id=AL6PEJZQVEB4XIME62LE42OIT7IOMRKCVSSMHQ2RH4EPBRPSBC5Q, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.321954332 (554h54m36.193037377s) [noise=true , address=145.79.227.36:4519 , piece_id=OBQSHCMXZXDYWO5ZQ22F7SZFBFZ55DURP6TCTSY533YV7PD2G66A, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.333499867 (554h54m36.204582902s) [noise=true , address=145.79.227.19:4077 , piece_id=MEDDJAJRK7GFKCRET4JIO4UJI6UW4WPG2PJ5E6CPEDVIFD6DOONA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.341732713 (554h54m36.212815749s) [noise=true , address=145.79.227.52:4445 , piece_id=4KD2OETYNY2QF53CWU3P5SVVXX4WADMR7FNR65S5O6RVW5T2TMLA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.343215437 (554h54m36.214298472s) [noise=true , address=145.79.227.11:4732 , piece_id=ED2TFFO4XNROP3CVF7TDS7WLQEZVLFK7VGDLPST73ZRFIUEXFMZA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.491781633 (554h54m36.362864668s) [noise=true , address=145.79.227.12:4765 , piece_id=HK7SHSNCV7JB34E3HPHSHMN7UPKUTAZGESJYAH2TMXQPI3QUH57A, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.517957315 (554h54m36.38904035s) [noise=true , address=45.91.116.12:4228 , piece_id=SMOYJWFDTCBDAMBZVNBO5RLV7UPOGYYVF7QXOXKZZGITZOXIIE6A, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.588144384 (554h54m36.459227429s) [noise=true , address=45.91.116.8:4046 , piece_id=HURGUSAGZEJX2GEUBLVWZ4B2FGM4TRF6CB7VTCOYAXTSSZYUZJNQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.601871713 (554h54m36.472954758s) [noise=true , address=77.90.31.19:4841 , piece_id=C4CUPNTQATSUMO26I3I2A4NW65ASY6EJUPR4A767FMJFBBYNLNMQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.616576253 (554h54m36.487659288s) [noise=true , address=45.91.116.4:4939 , piece_id=OKJG2NIUUNIROT7B5FZQCO2CECTCJRGVLPDWWTJ45QD4TKDM66IQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.937330425 (554h54m36.80841346s) [noise=true , address=45.91.116.6:4975 , piece_id=SZZNUDKG3XP6XED2ZI2TZ2ESVNZ5K3DVEEEEVPFSOSZG5DCIP4ZQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: <nil> Dec 28 18:23:36.937352146 (554h54m36.808435192s) [seg_pos={0 0}, upload=486493]: Segment reached optimal threshold of 13 pieces. Dec 28 18:23:36.937561114 (554h54m36.808644149s) [noise=true , address=145.79.227.4:4185 , piece_id=QXZHEH2MBJ5V7UGV3HOIUXJCB5QU2Z3YX4UYPQETQLY3BR76WTIA, seg_pos={0 0}, upload=486493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:60318->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:60318->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60318->145.79.227.4:4185: use of closed network connection Dec 28 18:23:36.937569139 (554h54m36.808652175s) [noise=true , address=145.79.227.4:4185 , piece_id=QXZHEH2MBJ5V7UGV3HOIUXJCB5QU2Z3YX4UYPQETQLY3BR76WTIA, seg_pos={0 0}, upload=486493]: Long tail cancellation due to optimal threshold Dec 28 18:23:36.937619125 (554h54m36.80870216s) [noise=true , address=77.90.31.18:4808 , piece_id=3BIJMDJ5GSBWPFQIVAVO74O7BSJFCMNKEHYAF4HUNSXA726F7VLQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 18:23:36.937629044 (554h54m36.808712079s) [noise=true , address=77.90.31.18:4808 , piece_id=3BIJMDJ5GSBWPFQIVAVO74O7BSJFCMNKEHYAF4HUNSXA726F7VLQ, seg_pos={0 0}, upload=486493]: Long tail cancellation due to optimal threshold Dec 28 18:23:36.937969122 (554h54m36.809052157s) [noise=true , address=145.79.227.26:4653 , piece_id=PGPQ6JCA4M3N6K3MMTFIMMNY2FOKV36PHTWVDNLLDVZP2Z3BPSCQ, seg_pos={0 0}, upload=486493]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: write tcp 188.166.225.170:54790->145.79.227.26:4653: use of closed network connection; write tcp 188.166.225.170:54790->145.79.227.26:4653: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54790->145.79.227.26:4653: use of closed network connection Dec 28 18:23:36.937976586 (554h54m36.809059631s) [noise=true , address=145.79.227.26:4653 , piece_id=PGPQ6JCA4M3N6K3MMTFIMMNY2FOKV36PHTWVDNLLDVZP2Z3BPSCQ, seg_pos={0 0}, upload=486493]: Long tail cancellation due to optimal threshold Dec 28 18:23:36.938087917 (554h54m36.809170952s) [seg_pos={0 0}, upload=486493]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:23:36.938162369 (554h54m36.809245414s) [upload=486493]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:23:36.938167128 (554h54m36.809250163s) [upload=486493]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:23:37.224268686 (554h54m37.095351721s) [upload=486493]: Done uploading segments... Dec 28 18:23:37.224295466 (554h54m37.095378511s) [upload=486493]: Upload finished. err: <nil> Duration: 30.003100368s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 3.212853262s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:23:39 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.21287863s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X........X.............X.X......................] duration: 15.320804695s trace: https://jaeger.dev.storj.io/trace/5eda81bca04798b7 FAIL (2025-12-28 18:23:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794293523 (554h35m21.665376568s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794729272 (554h35m21.665812307s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795130186 (554h35m21.666213231s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795534857 (554h35m21.666617902s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795928286 (554h35m21.667011331s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796356250 (554h35m21.667439285s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796770529 (554h35m21.667853574s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797234142 (554h35m21.668317187s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797701582 (554h35m21.668784697s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798145758 (554h35m21.669228793s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798717497 (554h35m21.669800532s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799245331 (554h35m21.670328377s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799828242 (554h35m21.670911287s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.800407144 (554h35m21.671490189s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.801046071 (554h35m21.672129116s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:23.074697857 (554h35m22.945780892s) [upload=486222]: Next returned nil segment Dec 28 18:04:23.074711302 (554h35m22.945794337s) [upload=486222]: Waiting for error group managing segments... Dec 28 18:04:23.417354815 (554h35m23.28843785s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.486601663 (554h35m23.357684708s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.668116249 (554h35m23.539199284s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.843876887 (554h35m23.714959922s) [noise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.848651940 (554h35m23.719734985s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.875480346 (554h35m23.746563381s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.029712621 (554h35m23.900795666s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.428381026 (554h35m24.299464061s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.550713979 (554h35m24.421797024s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.561873249 (554h35m24.432956284s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.583710861 (554h35m24.454793896s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.596115952 (554h35m24.467198997s) [noise=true , address=145.79.227.29:4959 , piece_id=F4UQHHNYPJVFEF336W3V5HIEPBKRQWM4DKOJAEXVHGULYWHLK75A, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.630336734 (554h35m24.501419779s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681392994 (554h35m24.552476029s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681422491 (554h35m24.552505556s) [seg_pos={0 0}, upload=486222]: Segment reached optimal threshold of 14 pieces. Dec 28 18:04:24.681622883 (554h35m24.552705918s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection Dec 28 18:04:24.681629966 (554h35m24.552713011s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681732601 (554h35m24.552815646s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681742510 (554h35m24.552825555s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681912925 (554h35m24.55299596s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681919507 (554h35m24.553002542s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682191224 (554h35m24.55327427s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.682198829 (554h35m24.553281864s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682218246 (554h35m24.553301281s) [seg_pos={0 0}, upload=486222]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:04:24.682292608 (554h35m24.553375653s) [upload=486222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:04:24.682297497 (554h35m24.553380532s) [upload=486222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:04:24.980627610 (554h35m24.851710655s) [upload=486222]: Done uploading segments... Dec 28 18:04:24.980654070 (554h35m24.851737105s) [upload=486222]: Upload finished. err: <nil> Duration: 15.320830545s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..XX........X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.X] duration: 30.002413691s trace: https://jaeger.dev.storj.io/trace/239484ed3d428344 FAIL (2025-12-28 18:23:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: L5VMJN6IXDRLSTTD5WQLN4ZXCCVP45NRWGKGRH6TWRORAWAONEYA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: PODG2XYDGHLLIJBE4WJQGTWWWEWPMUXKQDKYBQ4BUDI6TBLKJ3SQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: Q7EL2J6RHKST7EH3TKS5H4QHJZJAAIKZJN2IK2CLRBH7CVOHMVDQ) context deadline exceeded ======LOGS====== Dec 28 18:22:41.950066255 (554h53m41.821149301s) [upload=486483]: Starting upload Dec 28 18:22:41.950160795 (554h53m41.82124384s) [upload=486483]: Done starting upload Dec 28 18:22:41.950206983 (554h53m41.821290018s) [upload=486483]: Uploading segments... Dec 28 18:22:41.950497957 (554h53m41.821580992s) [upload=486483]: Got next segment. Inline: false Dec 28 18:22:41.950558833 (554h53m41.821641878s) [upload=486483]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:22:41.950565255 (554h53m41.8216483s) [upload=486483]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:22:42.193750111 (554h53m42.064833156s) [seg_pos={0 0}, upload=486483]: Begin upload segment... Dec 28 18:22:42.193846475 (554h53m42.06492951s) [noise=true , address=77.90.31.21:4878 , piece_id=27JXBYJUEYETOYFK47WSRKAGFT3DDW3DOXNQGUSMC3R5IIQTGPYQ, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.193864400 (554h53m42.064947435s) [noise=true , address=45.91.116.12:4225 , piece_id=Q7EL2J6RHKST7EH3TKS5H4QHJZJAAIKZJN2IK2CLRBH7CVOHMVDQ, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.194337229 (554h53m42.065420275s) [noise=true , address=145.79.227.15:4873 , piece_id=22AEMUNPVDE6T3PCGFJDMBDXGNR2TDSCCYWR6UCJPAEPUUMPKRSQ, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.194826872 (554h53m42.065909907s) [noise=true , address=145.79.227.39:4015 , piece_id=GLSHJGTJ4ZJBR3DLPA4RL3XKYMIN4ACACA7XBCVWQTNEWXX3MHZQ, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.195360469 (554h53m42.066443504s) [noise=true , address=45.91.116.6:4974 , piece_id=NJ55IHR2JDT7Y34TXPYCI4C6ZIOGET6KTPK3BBMTRNEXAMHVZ3MQ, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.195816777 (554h53m42.066899812s) [noise=true , address=145.79.227.37:4565 , piece_id=YLAQ2KQ6DHUQDMT7HWLUPD6OSPQO76EWJKPCB4NY6Y3KAV3TY3KQ, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.196262226 (554h53m42.067345271s) [noise=true , address=45.91.116.11:4117 , piece_id=L4YF4AMZLGSQXJYC2XGNMAX6GQS2SYHQ6T24VBJD2C3XFLI5ZOUA, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.196679981 (554h53m42.067763026s) [noise=true , address=145.79.227.48:4301 , piece_id=HM4AO7RNXXPCBDZ4DHVYIHYKDGZRB77BWV7DZEKWUREJERYQS73A, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.197499301 (554h53m42.068582336s) [noise=true , address=45.91.116.5:4299 , piece_id=L5VMJN6IXDRLSTTD5WQLN4ZXCCVP45NRWGKGRH6TWRORAWAONEYA, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.198095736 (554h53m42.069178781s) [noise=true , address=45.91.116.14:4190 , piece_id=PODG2XYDGHLLIJBE4WJQGTWWWEWPMUXKQDKYBQ4BUDI6TBLKJ3SQ, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.198849111 (554h53m42.069932146s) [noise=true , address=77.90.31.14:4664 , piece_id=6K5T3ZDCDWYWY6MWBZZPR7EIT73L2UGOMZOL4LFKERKAPXKKMWUA, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.199744186 (554h53m42.070827221s) [noise=true , address=145.79.227.8:4329 , piece_id=VJV44HHW6VVR36PN6TB3SI377HOFUQU7WH5DY7F5ZPWYTY3CDB6Q, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.200116033 (554h53m42.071199068s) [seg_pos={0 0}, upload=486483]: Done begin upload segment. Dec 28 18:22:42.201286452 (554h53m42.072369487s) [noise=true , address=145.79.227.36:4519 , piece_id=MANH5QPB6VMWBJXPCOMCXJDFWMTGPXFE7ENL2WPMF3PC3GZTN7ZQ, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.202638687 (554h53m42.073721722s) [noise=true , address=145.79.227.22:4510 , piece_id=BNFOF5BFY25VWRZPFDPK2BXAEGD2K5VOY5GJRW3AT64RC3QHDKSQ, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.204163471 (554h53m42.075246506s) [noise=true , address=145.79.227.19:4077 , piece_id=LLSLPC3WB3RGDQBCPLHPMZIBL53T4RKU6VCPZFD6ESU25LPE6KFA, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:42.205876322 (554h53m42.076959368s) [noise=true , address=145.79.227.7:4293 , piece_id=BAB62ACMELS5XO7S4TYBGZ2N4V5T47ZTR5HHZERNRJQHKODJHGVA, seg_pos={0 0}, upload=486483]: Uploading piece... Dec 28 18:22:43.908360708 (554h53m43.779443743s) [upload=486483]: Next returned nil segment Dec 28 18:22:43.908378462 (554h53m43.779461497s) [upload=486483]: Waiting for error group managing segments... Dec 28 18:22:44.195696225 (554h53m44.06677926s) [noise=true , address=145.79.227.19:4077 , piece_id=LLSLPC3WB3RGDQBCPLHPMZIBL53T4RKU6VCPZFD6ESU25LPE6KFA, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.206908605 (554h53m44.07799164s) [noise=true , address=145.79.227.22:4510 , piece_id=BNFOF5BFY25VWRZPFDPK2BXAEGD2K5VOY5GJRW3AT64RC3QHDKSQ, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.207231650 (554h53m44.078314685s) [noise=true , address=145.79.227.7:4293 , piece_id=BAB62ACMELS5XO7S4TYBGZ2N4V5T47ZTR5HHZERNRJQHKODJHGVA, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.208083282 (554h53m44.079166317s) [noise=true , address=145.79.227.8:4329 , piece_id=VJV44HHW6VVR36PN6TB3SI377HOFUQU7WH5DY7F5ZPWYTY3CDB6Q, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.211440645 (554h53m44.08252368s) [noise=true , address=145.79.227.48:4301 , piece_id=HM4AO7RNXXPCBDZ4DHVYIHYKDGZRB77BWV7DZEKWUREJERYQS73A, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.218790149 (554h53m44.089873184s) [noise=true , address=145.79.227.36:4519 , piece_id=MANH5QPB6VMWBJXPCOMCXJDFWMTGPXFE7ENL2WPMF3PC3GZTN7ZQ, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.222461952 (554h53m44.093544997s) [noise=true , address=145.79.227.37:4565 , piece_id=YLAQ2KQ6DHUQDMT7HWLUPD6OSPQO76EWJKPCB4NY6Y3KAV3TY3KQ, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.223068897 (554h53m44.094151932s) [noise=true , address=145.79.227.39:4015 , piece_id=GLSHJGTJ4ZJBR3DLPA4RL3XKYMIN4ACACA7XBCVWQTNEWXX3MHZQ, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.276125686 (554h53m44.147208731s) [noise=true , address=145.79.227.15:4873 , piece_id=22AEMUNPVDE6T3PCGFJDMBDXGNR2TDSCCYWR6UCJPAEPUUMPKRSQ, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.425105830 (554h53m44.296188875s) [noise=true , address=45.91.116.12:4225 , piece_id=Q7EL2J6RHKST7EH3TKS5H4QHJZJAAIKZJN2IK2CLRBH7CVOHMVDQ, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.439345035 (554h53m44.31042807s) [noise=true , address=45.91.116.14:4190 , piece_id=PODG2XYDGHLLIJBE4WJQGTWWWEWPMUXKQDKYBQ4BUDI6TBLKJ3SQ, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.570257895 (554h53m44.44134094s) [noise=true , address=45.91.116.6:4974 , piece_id=NJ55IHR2JDT7Y34TXPYCI4C6ZIOGET6KTPK3BBMTRNEXAMHVZ3MQ, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.573674591 (554h53m44.444757636s) [noise=true , address=45.91.116.5:4299 , piece_id=L5VMJN6IXDRLSTTD5WQLN4ZXCCVP45NRWGKGRH6TWRORAWAONEYA, seg_pos={0 0}, upload=486483]: Done uploading piece. err: <nil> Dec 28 18:22:44.573689189 (554h53m44.444772234s) [seg_pos={0 0}, upload=486483]: Segment reached optimal threshold of 13 pieces. Dec 28 18:22:44.574067920 (554h53m44.445150955s) [noise=true , address=77.90.31.14:4664 , piece_id=6K5T3ZDCDWYWY6MWBZZPR7EIT73L2UGOMZOL4LFKERKAPXKKMWUA, seg_pos={0 0}, upload=486483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: read tcp 188.166.225.170:44162->77.90.31.14:4664: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 18:22:44.574077448 (554h53m44.445160494s) [noise=true , address=45.91.116.11:4117 , piece_id=L4YF4AMZLGSQXJYC2XGNMAX6GQS2SYHQ6T24VBJD2C3XFLI5ZOUA, seg_pos={0 0}, upload=486483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 28 18:22:44.574095753 (554h53m44.445178818s) [noise=true , address=45.91.116.11:4117 , piece_id=L4YF4AMZLGSQXJYC2XGNMAX6GQS2SYHQ6T24VBJD2C3XFLI5ZOUA, seg_pos={0 0}, upload=486483]: Long tail cancellation due to optimal threshold Dec 28 18:22:44.574237823 (554h53m44.445320868s) [noise=true , address=77.90.31.14:4664 , piece_id=6K5T3ZDCDWYWY6MWBZZPR7EIT73L2UGOMZOL4LFKERKAPXKKMWUA, seg_pos={0 0}, upload=486483]: Long tail cancellation due to optimal threshold Dec 28 18:22:44.574273301 (554h53m44.445356336s) [noise=true , address=77.90.31.21:4878 , piece_id=27JXBYJUEYETOYFK47WSRKAGFT3DDW3DOXNQGUSMC3R5IIQTGPYQ, seg_pos={0 0}, upload=486483]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: read tcp 188.166.225.170:55392->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 18:22:44.574290574 (554h53m44.445373619s) [noise=true , address=77.90.31.21:4878 , piece_id=27JXBYJUEYETOYFK47WSRKAGFT3DDW3DOXNQGUSMC3R5IIQTGPYQ, seg_pos={0 0}, upload=486483]: Long tail cancellation due to optimal threshold Dec 28 18:22:44.574310451 (554h53m44.445393486s) [seg_pos={0 0}, upload=486483]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:22:44.574451169 (554h53m44.445534204s) [upload=486483]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:22:44.574460628 (554h53m44.445543673s) [upload=486483]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:22:44.863413214 (554h53m44.734496249s) [upload=486483]: Done uploading segments... Dec 28 18:22:44.863431649 (554h53m44.734514694s) [upload=486483]: Upload finished. err: <nil> Duration: 30.002436865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X........X.............X.X.....................] duration: 40.345256171s trace: https://jaeger.dev.storj.io/trace/5eda81bca04798b7 FAIL (2025-12-28 18:22:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794293523 (554h35m21.665376568s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794729272 (554h35m21.665812307s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795130186 (554h35m21.666213231s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795534857 (554h35m21.666617902s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795928286 (554h35m21.667011331s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796356250 (554h35m21.667439285s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796770529 (554h35m21.667853574s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797234142 (554h35m21.668317187s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797701582 (554h35m21.668784697s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798145758 (554h35m21.669228793s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798717497 (554h35m21.669800532s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799245331 (554h35m21.670328377s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799828242 (554h35m21.670911287s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.800407144 (554h35m21.671490189s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.801046071 (554h35m21.672129116s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:23.074697857 (554h35m22.945780892s) [upload=486222]: Next returned nil segment Dec 28 18:04:23.074711302 (554h35m22.945794337s) [upload=486222]: Waiting for error group managing segments... Dec 28 18:04:23.417354815 (554h35m23.28843785s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.486601663 (554h35m23.357684708s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.668116249 (554h35m23.539199284s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.843876887 (554h35m23.714959922s) [noise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.848651940 (554h35m23.719734985s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.875480346 (554h35m23.746563381s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.029712621 (554h35m23.900795666s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.428381026 (554h35m24.299464061s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.550713979 (554h35m24.421797024s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.561873249 (554h35m24.432956284s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.583710861 (554h35m24.454793896s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.596115952 (554h35m24.467198997s) [noise=true , address=145.79.227.29:4959 , piece_id=F4UQHHNYPJVFEF336W3V5HIEPBKRQWM4DKOJAEXVHGULYWHLK75A, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.630336734 (554h35m24.501419779s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681392994 (554h35m24.552476029s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681422491 (554h35m24.552505556s) [seg_pos={0 0}, upload=486222]: Segment reached optimal threshold of 14 pieces. Dec 28 18:04:24.681622883 (554h35m24.552705918s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection Dec 28 18:04:24.681629966 (554h35m24.552713011s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681732601 (554h35m24.552815646s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681742510 (554h35m24.552825555s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681912925 (554h35m24.55299596s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681919507 (554h35m24.553002542s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682191224 (554h35m24.55327427s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.682198829 (554h35m24.553281864s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682218246 (554h35m24.553301281s) [seg_pos={0 0}, upload=486222]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:04:24.682292608 (554h35m24.553375653s) [upload=486222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:04:24.682297497 (554h35m24.553380532s) [upload=486222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:04:24.980627610 (554h35m24.851710655s) [upload=486222]: Done uploading segments... Dec 28 18:04:24.980654070 (554h35m24.851737105s) [upload=486222]: Upload finished. err: <nil> Duration: 40.345278424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.491023785s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:22:40 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.491062339s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 3.75541298s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:22:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.755442345s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X........X.............X.X....................] duration: 48.334667088s trace: https://jaeger.dev.storj.io/trace/5eda81bca04798b7 FAIL (2025-12-28 18:21:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794293523 (554h35m21.665376568s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794729272 (554h35m21.665812307s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795130186 (554h35m21.666213231s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795534857 (554h35m21.666617902s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795928286 (554h35m21.667011331s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796356250 (554h35m21.667439285s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796770529 (554h35m21.667853574s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797234142 (554h35m21.668317187s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797701582 (554h35m21.668784697s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798145758 (554h35m21.669228793s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798717497 (554h35m21.669800532s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799245331 (554h35m21.670328377s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799828242 (554h35m21.670911287s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.800407144 (554h35m21.671490189s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.801046071 (554h35m21.672129116s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:23.074697857 (554h35m22.945780892s) [upload=486222]: Next returned nil segment Dec 28 18:04:23.074711302 (554h35m22.945794337s) [upload=486222]: Waiting for error group managing segments... Dec 28 18:04:23.417354815 (554h35m23.28843785s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.486601663 (554h35m23.357684708s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.668116249 (554h35m23.539199284s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.843876887 (554h35m23.714959922s) [noise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.848651940 (554h35m23.719734985s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.875480346 (554h35m23.746563381s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.029712621 (554h35m23.900795666s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.428381026 (554h35m24.299464061s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.550713979 (554h35m24.421797024s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.561873249 (554h35m24.432956284s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.583710861 (554h35m24.454793896s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.596115952 (554h35m24.467198997s) [noise=true , address=145.79.227.29:4959 , piece_id=F4UQHHNYPJVFEF336W3V5HIEPBKRQWM4DKOJAEXVHGULYWHLK75A, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.630336734 (554h35m24.501419779s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681392994 (554h35m24.552476029s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681422491 (554h35m24.552505556s) [seg_pos={0 0}, upload=486222]: Segment reached optimal threshold of 14 pieces. Dec 28 18:04:24.681622883 (554h35m24.552705918s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection Dec 28 18:04:24.681629966 (554h35m24.552713011s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681732601 (554h35m24.552815646s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681742510 (554h35m24.552825555s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681912925 (554h35m24.55299596s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681919507 (554h35m24.553002542s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682191224 (554h35m24.55327427s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.682198829 (554h35m24.553281864s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682218246 (554h35m24.553301281s) [seg_pos={0 0}, upload=486222]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:04:24.682292608 (554h35m24.553375653s) [upload=486222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:04:24.682297497 (554h35m24.553380532s) [upload=486222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:04:24.980627610 (554h35m24.851710655s) [upload=486222]: Done uploading segments... Dec 28 18:04:24.980654070 (554h35m24.851737105s) [upload=486222]: Upload finished. err: <nil> Duration: 48.334693707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..XX........X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X.] duration: 5.776168621s trace: https://jaeger.dev.storj.io/trace/27c71b2e24780481 FAIL (2025-12-28 18:21:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 18:21:03.139451808 (554h52m3.010534853s) [upload=486460]: Starting upload Dec 28 18:21:03.139530307 (554h52m3.010613352s) [upload=486460]: Done starting upload Dec 28 18:21:03.139644254 (554h52m3.010727329s) [upload=486460]: Uploading segments... Dec 28 18:21:03.139910722 (554h52m3.010993767s) [upload=486460]: Got next segment. Inline: false Dec 28 18:21:03.139969133 (554h52m3.011052168s) [upload=486460]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:21:03.139975415 (554h52m3.01105846s) [upload=486460]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:21:03.385079516 (554h52m3.256162551s) [seg_pos={0 0}, upload=486460]: Begin upload segment... Dec 28 18:21:03.385140422 (554h52m3.256223457s) [seg_pos={0 0}, upload=486460]: Done begin upload segment. Dec 28 18:21:03.385188393 (554h52m3.256271438s) [noise=true , address=77.90.31.12:4589 , piece_id=KPY2JLHMRP46JY3KG33B6WH2VWCLCOZAP6H2O2IF3M44ULSIHDMQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.385357755 (554h52m3.256440801s) [noise=true , address=77.90.31.14:4661 , piece_id=53PDGLH43KE5BQONDWXZSNK3JECVZMNCEUMVB6ZQQF4TWOOXPWTQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.386043782 (554h52m3.257126837s) [noise=true , address=45.91.116.7:4010 , piece_id=L4OKY4NIXJQG4BTF6MKFXDF3X4TWCXFXW2JF2FKO4LY3I2YFOJDA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.386736531 (554h52m3.257819586s) [noise=true , address=145.79.227.14:4837 , piece_id=35KTLPHBWV6VQN7EJPWAQPWAFZ4YNS5JAXYC767CNDHRKRHX675A, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.386768813 (554h52m3.257851898s) [noise=true , address=45.91.116.12:4228 , piece_id=25OIOM27E2GOGW53VZH7IOASPSNT634FWOHMZ5EFMYH4XM6JUJLQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.387283854 (554h52m3.258366909s) [noise=true , address=145.79.227.15:4874 , piece_id=6W4XO7SIZ3SE5APDVGQSGCQ3WCNIKHIHRW6C3WWE7DVDJM4JDSTQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.387364156 (554h52m3.258447201s) [noise=true , address=145.79.227.19:4077 , piece_id=KNILQ2XNKBKM52Q3GDCW2EFNW3GUDLZBCE5N7WCZAWTSHX4TCRYA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.387733250 (554h52m3.258816295s) [noise=true , address=77.90.31.17:4770 , piece_id=SKCNMKBGDJ5J6GRNZC7LDRC4VLUDPATJGYUYKIWPZ3WWPH4W473A, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.387884758 (554h52m3.258967803s) [noise=true , address=145.79.227.41:4087 , piece_id=JVWI7MFDRQE2JEJUVOKVSSNPA2SBFHSM5G46YW2WBTNSQIBSHAZA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.388192083 (554h52m3.259275118s) [noise=true , address=145.79.227.47:4267 , piece_id=WOCHABFVMHXBR4WV6EPPJYT3RQJES4LYVTBEW3H4E2RWZSGKTAIA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.388327301 (554h52m3.259410336s) [noise=true , address=45.91.116.10:4082 , piece_id=7MS26K5NN6C5UYBGSJQQLFCDFTTXYWSU5YMOVCLI3IYECEEW5WGA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.388698888 (554h52m3.259781933s) [noise=true , address=145.79.227.22:4510 , piece_id=3CQSKDDZZJJTT3VKUKQKK3MY54NKYDD2XGWOKPBOHJJRQUGKFKVQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.388755576 (554h52m3.259838621s) [noise=true , address=45.91.116.11:4120 , piece_id=VYPOJCSGJSE6MN372E35QHVDU72QOOKPNVO5Z4NONWRHKRVDJKSQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.389189131 (554h52m3.260272176s) [noise=true , address=145.79.227.3:4150 , piece_id=BFHKSMFVB26JJOEWNHSWF7UR2AOVHZ225YX4JAF3NL7AXJZPS4GA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.389268653 (554h52m3.260351698s) [noise=true , address=45.91.116.5:4300 , piece_id=BVBOX5MRZ5GW5LGQSPVQS56A7I3YC2PBEDOQW4MF4MSVOYPWT4IQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.389753286 (554h52m3.260836331s) [noise=true , address=77.90.31.13:4627 , piece_id=2RFMOAUVSNZR756ACJC3CS5EZ2XSJRGD6WFWY6F7TZ5JKQDZDHNQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:05.016072179 (554h52m4.887155234s) [upload=486460]: Next returned nil segment Dec 28 18:21:05.016124680 (554h52m4.887207735s) [upload=486460]: Waiting for error group managing segments... Dec 28 18:21:05.394910403 (554h52m5.265993448s) [noise=true , address=145.79.227.19:4077 , piece_id=KNILQ2XNKBKM52Q3GDCW2EFNW3GUDLZBCE5N7WCZAWTSHX4TCRYA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.396834637 (554h52m5.267917683s) [noise=true , address=145.79.227.22:4510 , piece_id=3CQSKDDZZJJTT3VKUKQKK3MY54NKYDD2XGWOKPBOHJJRQUGKFKVQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.398577015 (554h52m5.26966006s) [noise=true , address=145.79.227.15:4874 , piece_id=6W4XO7SIZ3SE5APDVGQSGCQ3WCNIKHIHRW6C3WWE7DVDJM4JDSTQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.401475925 (554h52m5.27255897s) [noise=true , address=145.79.227.3:4150 , piece_id=BFHKSMFVB26JJOEWNHSWF7UR2AOVHZ225YX4JAF3NL7AXJZPS4GA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.423264644 (554h52m5.29434769s) [noise=true , address=145.79.227.41:4087 , piece_id=JVWI7MFDRQE2JEJUVOKVSSNPA2SBFHSM5G46YW2WBTNSQIBSHAZA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.429773509 (554h52m5.300856564s) [noise=true , address=145.79.227.47:4267 , piece_id=WOCHABFVMHXBR4WV6EPPJYT3RQJES4LYVTBEW3H4E2RWZSGKTAIA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.441117980 (554h52m5.312201026s) [noise=true , address=145.79.227.14:4837 , piece_id=35KTLPHBWV6VQN7EJPWAQPWAFZ4YNS5JAXYC767CNDHRKRHX675A, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.645794731 (554h52m5.516877786s) [noise=true , address=45.91.116.11:4120 , piece_id=VYPOJCSGJSE6MN372E35QHVDU72QOOKPNVO5Z4NONWRHKRVDJKSQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.662551799 (554h52m5.533634844s) [noise=true , address=45.91.116.12:4228 , piece_id=25OIOM27E2GOGW53VZH7IOASPSNT634FWOHMZ5EFMYH4XM6JUJLQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.674945699 (554h52m5.546028744s) [noise=true , address=77.90.31.12:4589 , piece_id=KPY2JLHMRP46JY3KG33B6WH2VWCLCOZAP6H2O2IF3M44ULSIHDMQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.682786510 (554h52m5.553869545s) [noise=true , address=77.90.31.14:4661 , piece_id=53PDGLH43KE5BQONDWXZSNK3JECVZMNCEUMVB6ZQQF4TWOOXPWTQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.687044127 (554h52m5.558127162s) [noise=true , address=77.90.31.17:4770 , piece_id=SKCNMKBGDJ5J6GRNZC7LDRC4VLUDPATJGYUYKIWPZ3WWPH4W473A, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.703723026 (554h52m5.574806071s) [noise=true , address=45.91.116.7:4010 , piece_id=L4OKY4NIXJQG4BTF6MKFXDF3X4TWCXFXW2JF2FKO4LY3I2YFOJDA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.703749607 (554h52m5.574832652s) [seg_pos={0 0}, upload=486460]: Segment reached optimal threshold of 13 pieces. Dec 28 18:21:05.703983042 (554h52m5.575066087s) [noise=true , address=77.90.31.13:4627 , piece_id=2RFMOAUVSNZR756ACJC3CS5EZ2XSJRGD6WFWY6F7TZ5JKQDZDHNQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:48470->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:48470->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48470->77.90.31.13:4627: use of closed network connection Dec 28 18:21:05.703991788 (554h52m5.575074833s) [noise=true , address=77.90.31.13:4627 , piece_id=2RFMOAUVSNZR756ACJC3CS5EZ2XSJRGD6WFWY6F7TZ5JKQDZDHNQ, seg_pos={0 0}, upload=486460]: Long tail cancellation due to optimal threshold Dec 28 18:21:05.704253736 (554h52m5.575336781s) [noise=true , address=45.91.116.5:4300 , piece_id=BVBOX5MRZ5GW5LGQSPVQS56A7I3YC2PBEDOQW4MF4MSVOYPWT4IQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 18:21:05.704264297 (554h52m5.575347332s) [noise=true , address=45.91.116.5:4300 , piece_id=BVBOX5MRZ5GW5LGQSPVQS56A7I3YC2PBEDOQW4MF4MSVOYPWT4IQ, seg_pos={0 0}, upload=486460]: Long tail cancellation due to optimal threshold Dec 28 18:21:05.704279957 (554h52m5.575363122s) [noise=true , address=45.91.116.10:4082 , piece_id=7MS26K5NN6C5UYBGSJQQLFCDFTTXYWSU5YMOVCLI3IYECEEW5WGA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 18:21:05.704289945 (554h52m5.57537298s) [noise=true , address=45.91.116.10:4082 , piece_id=7MS26K5NN6C5UYBGSJQQLFCDFTTXYWSU5YMOVCLI3IYECEEW5WGA, seg_pos={0 0}, upload=486460]: Long tail cancellation due to optimal threshold Dec 28 18:21:05.704428209 (554h52m5.575511254s) [seg_pos={0 0}, upload=486460]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:21:05.704522238 (554h52m5.575605283s) [upload=486460]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:21:05.704529111 (554h52m5.575612156s) [upload=486460]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:21:05.992878579 (554h52m5.863961624s) [upload=486460]: Done uploading segments... Dec 28 18:21:05.992906292 (554h52m5.863989337s) [upload=486460]: Upload finished. err: <nil> Duration: 5.776195243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X..XX........X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.X] duration: 30.002829181s trace: https://jaeger.dev.storj.io/trace/27c71b2e24780481 FAIL (2025-12-28 18:21:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 18:21:03.139451808 (554h52m3.010534853s) [upload=486460]: Starting upload Dec 28 18:21:03.139530307 (554h52m3.010613352s) [upload=486460]: Done starting upload Dec 28 18:21:03.139644254 (554h52m3.010727329s) [upload=486460]: Uploading segments... Dec 28 18:21:03.139910722 (554h52m3.010993767s) [upload=486460]: Got next segment. Inline: false Dec 28 18:21:03.139969133 (554h52m3.011052168s) [upload=486460]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:21:03.139975415 (554h52m3.01105846s) [upload=486460]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:21:03.385079516 (554h52m3.256162551s) [seg_pos={0 0}, upload=486460]: Begin upload segment... Dec 28 18:21:03.385140422 (554h52m3.256223457s) [seg_pos={0 0}, upload=486460]: Done begin upload segment. Dec 28 18:21:03.385188393 (554h52m3.256271438s) [noise=true , address=77.90.31.12:4589 , piece_id=KPY2JLHMRP46JY3KG33B6WH2VWCLCOZAP6H2O2IF3M44ULSIHDMQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.385357755 (554h52m3.256440801s) [noise=true , address=77.90.31.14:4661 , piece_id=53PDGLH43KE5BQONDWXZSNK3JECVZMNCEUMVB6ZQQF4TWOOXPWTQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.386043782 (554h52m3.257126837s) [noise=true , address=45.91.116.7:4010 , piece_id=L4OKY4NIXJQG4BTF6MKFXDF3X4TWCXFXW2JF2FKO4LY3I2YFOJDA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.386736531 (554h52m3.257819586s) [noise=true , address=145.79.227.14:4837 , piece_id=35KTLPHBWV6VQN7EJPWAQPWAFZ4YNS5JAXYC767CNDHRKRHX675A, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.386768813 (554h52m3.257851898s) [noise=true , address=45.91.116.12:4228 , piece_id=25OIOM27E2GOGW53VZH7IOASPSNT634FWOHMZ5EFMYH4XM6JUJLQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.387283854 (554h52m3.258366909s) [noise=true , address=145.79.227.15:4874 , piece_id=6W4XO7SIZ3SE5APDVGQSGCQ3WCNIKHIHRW6C3WWE7DVDJM4JDSTQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.387364156 (554h52m3.258447201s) [noise=true , address=145.79.227.19:4077 , piece_id=KNILQ2XNKBKM52Q3GDCW2EFNW3GUDLZBCE5N7WCZAWTSHX4TCRYA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.387733250 (554h52m3.258816295s) [noise=true , address=77.90.31.17:4770 , piece_id=SKCNMKBGDJ5J6GRNZC7LDRC4VLUDPATJGYUYKIWPZ3WWPH4W473A, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.387884758 (554h52m3.258967803s) [noise=true , address=145.79.227.41:4087 , piece_id=JVWI7MFDRQE2JEJUVOKVSSNPA2SBFHSM5G46YW2WBTNSQIBSHAZA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.388192083 (554h52m3.259275118s) [noise=true , address=145.79.227.47:4267 , piece_id=WOCHABFVMHXBR4WV6EPPJYT3RQJES4LYVTBEW3H4E2RWZSGKTAIA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.388327301 (554h52m3.259410336s) [noise=true , address=45.91.116.10:4082 , piece_id=7MS26K5NN6C5UYBGSJQQLFCDFTTXYWSU5YMOVCLI3IYECEEW5WGA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.388698888 (554h52m3.259781933s) [noise=true , address=145.79.227.22:4510 , piece_id=3CQSKDDZZJJTT3VKUKQKK3MY54NKYDD2XGWOKPBOHJJRQUGKFKVQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.388755576 (554h52m3.259838621s) [noise=true , address=45.91.116.11:4120 , piece_id=VYPOJCSGJSE6MN372E35QHVDU72QOOKPNVO5Z4NONWRHKRVDJKSQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.389189131 (554h52m3.260272176s) [noise=true , address=145.79.227.3:4150 , piece_id=BFHKSMFVB26JJOEWNHSWF7UR2AOVHZ225YX4JAF3NL7AXJZPS4GA, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.389268653 (554h52m3.260351698s) [noise=true , address=45.91.116.5:4300 , piece_id=BVBOX5MRZ5GW5LGQSPVQS56A7I3YC2PBEDOQW4MF4MSVOYPWT4IQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:03.389753286 (554h52m3.260836331s) [noise=true , address=77.90.31.13:4627 , piece_id=2RFMOAUVSNZR756ACJC3CS5EZ2XSJRGD6WFWY6F7TZ5JKQDZDHNQ, seg_pos={0 0}, upload=486460]: Uploading piece... Dec 28 18:21:05.016072179 (554h52m4.887155234s) [upload=486460]: Next returned nil segment Dec 28 18:21:05.016124680 (554h52m4.887207735s) [upload=486460]: Waiting for error group managing segments... Dec 28 18:21:05.394910403 (554h52m5.265993448s) [noise=true , address=145.79.227.19:4077 , piece_id=KNILQ2XNKBKM52Q3GDCW2EFNW3GUDLZBCE5N7WCZAWTSHX4TCRYA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.396834637 (554h52m5.267917683s) [noise=true , address=145.79.227.22:4510 , piece_id=3CQSKDDZZJJTT3VKUKQKK3MY54NKYDD2XGWOKPBOHJJRQUGKFKVQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.398577015 (554h52m5.26966006s) [noise=true , address=145.79.227.15:4874 , piece_id=6W4XO7SIZ3SE5APDVGQSGCQ3WCNIKHIHRW6C3WWE7DVDJM4JDSTQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.401475925 (554h52m5.27255897s) [noise=true , address=145.79.227.3:4150 , piece_id=BFHKSMFVB26JJOEWNHSWF7UR2AOVHZ225YX4JAF3NL7AXJZPS4GA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.423264644 (554h52m5.29434769s) [noise=true , address=145.79.227.41:4087 , piece_id=JVWI7MFDRQE2JEJUVOKVSSNPA2SBFHSM5G46YW2WBTNSQIBSHAZA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.429773509 (554h52m5.300856564s) [noise=true , address=145.79.227.47:4267 , piece_id=WOCHABFVMHXBR4WV6EPPJYT3RQJES4LYVTBEW3H4E2RWZSGKTAIA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.441117980 (554h52m5.312201026s) [noise=true , address=145.79.227.14:4837 , piece_id=35KTLPHBWV6VQN7EJPWAQPWAFZ4YNS5JAXYC767CNDHRKRHX675A, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.645794731 (554h52m5.516877786s) [noise=true , address=45.91.116.11:4120 , piece_id=VYPOJCSGJSE6MN372E35QHVDU72QOOKPNVO5Z4NONWRHKRVDJKSQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.662551799 (554h52m5.533634844s) [noise=true , address=45.91.116.12:4228 , piece_id=25OIOM27E2GOGW53VZH7IOASPSNT634FWOHMZ5EFMYH4XM6JUJLQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.674945699 (554h52m5.546028744s) [noise=true , address=77.90.31.12:4589 , piece_id=KPY2JLHMRP46JY3KG33B6WH2VWCLCOZAP6H2O2IF3M44ULSIHDMQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.682786510 (554h52m5.553869545s) [noise=true , address=77.90.31.14:4661 , piece_id=53PDGLH43KE5BQONDWXZSNK3JECVZMNCEUMVB6ZQQF4TWOOXPWTQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.687044127 (554h52m5.558127162s) [noise=true , address=77.90.31.17:4770 , piece_id=SKCNMKBGDJ5J6GRNZC7LDRC4VLUDPATJGYUYKIWPZ3WWPH4W473A, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.703723026 (554h52m5.574806071s) [noise=true , address=45.91.116.7:4010 , piece_id=L4OKY4NIXJQG4BTF6MKFXDF3X4TWCXFXW2JF2FKO4LY3I2YFOJDA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: <nil> Dec 28 18:21:05.703749607 (554h52m5.574832652s) [seg_pos={0 0}, upload=486460]: Segment reached optimal threshold of 13 pieces. Dec 28 18:21:05.703983042 (554h52m5.575066087s) [noise=true , address=77.90.31.13:4627 , piece_id=2RFMOAUVSNZR756ACJC3CS5EZ2XSJRGD6WFWY6F7TZ5JKQDZDHNQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:48470->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:48470->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48470->77.90.31.13:4627: use of closed network connection Dec 28 18:21:05.703991788 (554h52m5.575074833s) [noise=true , address=77.90.31.13:4627 , piece_id=2RFMOAUVSNZR756ACJC3CS5EZ2XSJRGD6WFWY6F7TZ5JKQDZDHNQ, seg_pos={0 0}, upload=486460]: Long tail cancellation due to optimal threshold Dec 28 18:21:05.704253736 (554h52m5.575336781s) [noise=true , address=45.91.116.5:4300 , piece_id=BVBOX5MRZ5GW5LGQSPVQS56A7I3YC2PBEDOQW4MF4MSVOYPWT4IQ, seg_pos={0 0}, upload=486460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 18:21:05.704264297 (554h52m5.575347332s) [noise=true , address=45.91.116.5:4300 , piece_id=BVBOX5MRZ5GW5LGQSPVQS56A7I3YC2PBEDOQW4MF4MSVOYPWT4IQ, seg_pos={0 0}, upload=486460]: Long tail cancellation due to optimal threshold Dec 28 18:21:05.704279957 (554h52m5.575363122s) [noise=true , address=45.91.116.10:4082 , piece_id=7MS26K5NN6C5UYBGSJQQLFCDFTTXYWSU5YMOVCLI3IYECEEW5WGA, seg_pos={0 0}, upload=486460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 18:21:05.704289945 (554h52m5.57537298s) [noise=true , address=45.91.116.10:4082 , piece_id=7MS26K5NN6C5UYBGSJQQLFCDFTTXYWSU5YMOVCLI3IYECEEW5WGA, seg_pos={0 0}, upload=486460]: Long tail cancellation due to optimal threshold Dec 28 18:21:05.704428209 (554h52m5.575511254s) [seg_pos={0 0}, upload=486460]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:21:05.704522238 (554h52m5.575605283s) [upload=486460]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:21:05.704529111 (554h52m5.575612156s) [upload=486460]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:21:05.992878579 (554h52m5.863961624s) [upload=486460]: Done uploading segments... Dec 28 18:21:05.992906292 (554h52m5.863989337s) [upload=486460]: Upload finished. err: <nil> Duration: 30.002858857s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.251223084s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:21:18 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.251251679s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 3.327566495s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:20:59 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.327593627s core uplink ap1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.540060555s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 18:20:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540091094s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X........X.............X.X...................] duration: 17.990301257s trace: https://jaeger.dev.storj.io/trace/5eda81bca04798b7 FAIL (2025-12-28 18:20:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794293523 (554h35m21.665376568s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794729272 (554h35m21.665812307s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795130186 (554h35m21.666213231s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795534857 (554h35m21.666617902s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795928286 (554h35m21.667011331s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796356250 (554h35m21.667439285s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796770529 (554h35m21.667853574s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797234142 (554h35m21.668317187s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797701582 (554h35m21.668784697s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798145758 (554h35m21.669228793s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798717497 (554h35m21.669800532s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799245331 (554h35m21.670328377s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799828242 (554h35m21.670911287s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.800407144 (554h35m21.671490189s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.801046071 (554h35m21.672129116s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:23.074697857 (554h35m22.945780892s) [upload=486222]: Next returned nil segment Dec 28 18:04:23.074711302 (554h35m22.945794337s) [upload=486222]: Waiting for error group managing segments... Dec 28 18:04:23.417354815 (554h35m23.28843785s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.486601663 (554h35m23.357684708s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.668116249 (554h35m23.539199284s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.843876887 (554h35m23.714959922s) [noise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.848651940 (554h35m23.719734985s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.875480346 (554h35m23.746563381s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.029712621 (554h35m23.900795666s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.428381026 (554h35m24.299464061s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.550713979 (554h35m24.421797024s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.561873249 (554h35m24.432956284s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.583710861 (554h35m24.454793896s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.596115952 (554h35m24.467198997s) [noise=true , address=145.79.227.29:4959 , piece_id=F4UQHHNYPJVFEF336W3V5HIEPBKRQWM4DKOJAEXVHGULYWHLK75A, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.630336734 (554h35m24.501419779s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681392994 (554h35m24.552476029s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681422491 (554h35m24.552505556s) [seg_pos={0 0}, upload=486222]: Segment reached optimal threshold of 14 pieces. Dec 28 18:04:24.681622883 (554h35m24.552705918s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection Dec 28 18:04:24.681629966 (554h35m24.552713011s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681732601 (554h35m24.552815646s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681742510 (554h35m24.552825555s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681912925 (554h35m24.55299596s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681919507 (554h35m24.553002542s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682191224 (554h35m24.55327427s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.682198829 (554h35m24.553281864s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682218246 (554h35m24.553301281s) [seg_pos={0 0}, upload=486222]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:04:24.682292608 (554h35m24.553375653s) [upload=486222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:04:24.682297497 (554h35m24.553380532s) [upload=486222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:04:24.980627610 (554h35m24.851710655s) [upload=486222]: Done uploading segments... Dec 28 18:04:24.980654070 (554h35m24.851737105s) [upload=486222]: Upload finished. err: <nil> Duration: 17.990332517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 3.273624273s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:20:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.273652767s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X..XX........X..XX.X.XX..X....XXX..X.X....X..X..X....X.X.] duration: 8.809178695s trace: https://jaeger.dev.storj.io/trace/5d0c1eb6e9125af0 FAIL (2025-12-28 18:20:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 18:19:21.637365805 (554h50m21.50844884s) [upload=486438]: Starting upload Dec 28 18:19:21.637462920 (554h50m21.508545965s) [upload=486438]: Done starting upload Dec 28 18:19:21.637579782 (554h50m21.508662837s) [upload=486438]: Uploading segments... Dec 28 18:19:21.637820741 (554h50m21.508903786s) [upload=486438]: Got next segment. Inline: false Dec 28 18:19:21.637869845 (554h50m21.50895288s) [upload=486438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:19:21.637875325 (554h50m21.50895836s) [upload=486438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:19:21.890229011 (554h50m21.761312056s) [seg_pos={0 0}, upload=486438]: Begin upload segment... Dec 28 18:19:21.890341735 (554h50m21.76142477s) [seg_pos={0 0}, upload=486438]: Done begin upload segment. Dec 28 18:19:21.890388835 (554h50m21.76147187s) [noise=true , address=145.79.227.8:4329 , piece_id=RBMYWNAOYFKOBNDQWKKHH3HSLTA6GG4OJTUCD5IZXU7OSYVZKOXA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.891366907 (554h50m21.762449952s) [noise=true , address=145.79.227.3:4151 , piece_id=WYULE7LOQW56VSUPLSH7OAJEWG7HGX6EFPXLDCXAERK32AWBXYGQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.891881808 (554h50m21.762964853s) [noise=true , address=45.91.116.13:4155 , piece_id=QRG3MK33XHMQFYNSHOHTD7RATCWKIAFZDADKXBICH62KDFONHH5A, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.892324781 (554h50m21.763407827s) [noise=true , address=145.79.227.13:4803 , piece_id=DSIMUZ3PU4N57KCIOSQ52IX7EAKXF4OJPKH2Q57QDM3SBBVJOWWQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.893661717 (554h50m21.764744752s) [noise=true , address=145.79.227.41:4085 , piece_id=T3WB3SUSGJIK47H7EWP3WWF6GAINCTDXX3D33TDCZBL6VTFDRICQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.894123848 (554h50m21.765206893s) [noise=true , address=145.79.227.15:4873 , piece_id=Y5UZBTE2WYA7TD4H6EO4G35OXGV5PP32ST5Q4EM4RNGAUDV53TZA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.894548887 (554h50m21.765631932s) [noise=true , address=145.79.227.51:4410 , piece_id=GSQD6EJ237UUDIPY6CMGNH3SYBWQ63I66PUMYWACIS6SLQEYMGTQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.895242227 (554h50m21.766325272s) [noise=true , address=77.90.31.12:4590 , piece_id=TNQACGMQ223MB7H6EQDGQN3AFEECVTKFNYVFZ5LKNO5LPTT4HCKA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.895902985 (554h50m21.76698602s) [noise=true , address=145.79.227.19:4078 , piece_id=2KL2PML5I7QFJV5HWN3YRJSZYUQKZ53I67G4VRJBRTGJG62YXJKQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.896326401 (554h50m21.767409446s) [noise=true , address=145.79.227.26:4653 , piece_id=BQQLUK4SPHPCY5PDEEJSRBPK53BVW22MSY7V6AZ5GKE7RL2PY2JA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.896841242 (554h50m21.767924277s) [noise=true , address=45.91.116.11:4119 , piece_id=6YMLRYCO5TBOHI4ZHVSPKDIVQKHY4BGLNNMA6ZWNBT2UTQHGWB6A, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.897294275 (554h50m21.76837732s) [noise=true , address=145.79.227.33:4869 , piece_id=ENMH6OW6NBEN7IQLLIAEWD5LJBXXWUWQMOWMQNNUXVFFJ2YDC4RA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.897656704 (554h50m21.768739749s) [noise=true , address=145.79.227.39:4013 , piece_id=CZVC4FGKXWAFHGBOBUKVEEOM54ZRIVIRSFOPKSFD2RESRVS6L26A, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.898158419 (554h50m21.769241464s) [noise=true , address=77.90.31.10:4519 , piece_id=CCAK3MGMMDICKYB5WYHV446SO5S3ZMMVOJ73FICJ75FN4TLBDPWA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.898581796 (554h50m21.769664841s) [noise=true , address=77.90.31.16:4736 , piece_id=OLS257CVX2OZ4PEIGXO5P252YTNO3HSCXKBO3ZO3COTAFF42C6AQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.898922855 (554h50m21.7700059s) [noise=true , address=145.79.227.11:4730 , piece_id=LATJLB3Z3MAPI6QPZAUYFU5NMWMKTZF3LDQPTC2MM7IULEAV6W4A, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:23.500839465 (554h50m23.37192252s) [upload=486438]: Next returned nil segment Dec 28 18:19:23.500855124 (554h50m23.371938159s) [upload=486438]: Waiting for error group managing segments... Dec 28 18:19:23.881284027 (554h50m23.752367062s) [noise=true , address=145.79.227.19:4078 , piece_id=2KL2PML5I7QFJV5HWN3YRJSZYUQKZ53I67G4VRJBRTGJG62YXJKQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.889296886 (554h50m23.760379931s) [noise=true , address=145.79.227.41:4085 , piece_id=T3WB3SUSGJIK47H7EWP3WWF6GAINCTDXX3D33TDCZBL6VTFDRICQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.891426821 (554h50m23.762509866s) [noise=true , address=145.79.227.3:4151 , piece_id=WYULE7LOQW56VSUPLSH7OAJEWG7HGX6EFPXLDCXAERK32AWBXYGQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.896719018 (554h50m23.767802063s) [noise=true , address=145.79.227.15:4873 , piece_id=Y5UZBTE2WYA7TD4H6EO4G35OXGV5PP32ST5Q4EM4RNGAUDV53TZA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.898127370 (554h50m23.769210405s) [noise=true , address=145.79.227.26:4653 , piece_id=BQQLUK4SPHPCY5PDEEJSRBPK53BVW22MSY7V6AZ5GKE7RL2PY2JA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.901860960 (554h50m23.772944005s) [noise=true , address=145.79.227.51:4410 , piece_id=GSQD6EJ237UUDIPY6CMGNH3SYBWQ63I66PUMYWACIS6SLQEYMGTQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.910762740 (554h50m23.781845775s) [noise=true , address=145.79.227.39:4013 , piece_id=CZVC4FGKXWAFHGBOBUKVEEOM54ZRIVIRSFOPKSFD2RESRVS6L26A, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.911464987 (554h50m23.782548022s) [noise=true , address=145.79.227.8:4329 , piece_id=RBMYWNAOYFKOBNDQWKKHH3HSLTA6GG4OJTUCD5IZXU7OSYVZKOXA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.914396029 (554h50m23.785479064s) [noise=true , address=145.79.227.11:4730 , piece_id=LATJLB3Z3MAPI6QPZAUYFU5NMWMKTZF3LDQPTC2MM7IULEAV6W4A, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.944864977 (554h50m23.815948022s) [noise=true , address=145.79.227.13:4803 , piece_id=DSIMUZ3PU4N57KCIOSQ52IX7EAKXF4OJPKH2Q57QDM3SBBVJOWWQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:24.126100581 (554h50m23.997183626s) [noise=true , address=45.91.116.11:4119 , piece_id=6YMLRYCO5TBOHI4ZHVSPKDIVQKHY4BGLNNMA6ZWNBT2UTQHGWB6A, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:24.201780459 (554h50m24.072863504s) [noise=true , address=77.90.31.16:4736 , piece_id=OLS257CVX2OZ4PEIGXO5P252YTNO3HSCXKBO3ZO3COTAFF42C6AQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:24.514648785 (554h50m24.38573182s) [noise=true , address=77.90.31.10:4519 , piece_id=CCAK3MGMMDICKYB5WYHV446SO5S3ZMMVOJ73FICJ75FN4TLBDPWA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:24.514669073 (554h50m24.385752118s) [seg_pos={0 0}, upload=486438]: Segment reached optimal threshold of 13 pieces. Dec 28 18:19:24.514875025 (554h50m24.385958061s) [noise=true , address=145.79.227.33:4869 , piece_id=ENMH6OW6NBEN7IQLLIAEWD5LJBXXWUWQMOWMQNNUXVFFJ2YDC4RA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 18:19:24.514882840 (554h50m24.385965875s) [noise=true , address=145.79.227.33:4869 , piece_id=ENMH6OW6NBEN7IQLLIAEWD5LJBXXWUWQMOWMQNNUXVFFJ2YDC4RA, seg_pos={0 0}, upload=486438]: Long tail cancellation due to optimal threshold Dec 28 18:19:24.514993741 (554h50m24.386076796s) [noise=true , address=77.90.31.12:4590 , piece_id=TNQACGMQ223MB7H6EQDGQN3AFEECVTKFNYVFZ5LKNO5LPTT4HCKA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 18:19:24.515004823 (554h50m24.386087868s) [noise=true , address=77.90.31.12:4590 , piece_id=TNQACGMQ223MB7H6EQDGQN3AFEECVTKFNYVFZ5LKNO5LPTT4HCKA, seg_pos={0 0}, upload=486438]: Long tail cancellation due to optimal threshold Dec 28 18:19:24.515297861 (554h50m24.386380906s) [noise=true , address=45.91.116.13:4155 , piece_id=QRG3MK33XHMQFYNSHOHTD7RATCWKIAFZDADKXBICH62KDFONHH5A, seg_pos={0 0}, upload=486438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:37338->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:37338->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: context canceled Dec 28 18:19:24.515305435 (554h50m24.38638847s) [noise=true , address=45.91.116.13:4155 , piece_id=QRG3MK33XHMQFYNSHOHTD7RATCWKIAFZDADKXBICH62KDFONHH5A, seg_pos={0 0}, upload=486438]: Long tail cancellation due to optimal threshold Dec 28 18:19:24.515321265 (554h50m24.38640431s) [seg_pos={0 0}, upload=486438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:19:24.515391148 (554h50m24.386474184s) [upload=486438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:19:24.515414183 (554h50m24.386497218s) [upload=486438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:19:24.798128851 (554h50m24.669211916s) [upload=486438]: Done uploading segments... Dec 28 18:19:24.798153016 (554h50m24.669236051s) [upload=486438]: Upload finished. err: <nil> Duration: 8.809217019s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.529682682s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 18:20:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.529737737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX.X..XX........X..XX.X.XX..X....XXX..X.X....X..X..X....X.X] duration: 30.003945455s trace: https://jaeger.dev.storj.io/trace/5d0c1eb6e9125af0 FAIL (2025-12-28 18:19:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 18:19:21.637365805 (554h50m21.50844884s) [upload=486438]: Starting upload Dec 28 18:19:21.637462920 (554h50m21.508545965s) [upload=486438]: Done starting upload Dec 28 18:19:21.637579782 (554h50m21.508662837s) [upload=486438]: Uploading segments... Dec 28 18:19:21.637820741 (554h50m21.508903786s) [upload=486438]: Got next segment. Inline: false Dec 28 18:19:21.637869845 (554h50m21.50895288s) [upload=486438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:19:21.637875325 (554h50m21.50895836s) [upload=486438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:19:21.890229011 (554h50m21.761312056s) [seg_pos={0 0}, upload=486438]: Begin upload segment... Dec 28 18:19:21.890341735 (554h50m21.76142477s) [seg_pos={0 0}, upload=486438]: Done begin upload segment. Dec 28 18:19:21.890388835 (554h50m21.76147187s) [noise=true , address=145.79.227.8:4329 , piece_id=RBMYWNAOYFKOBNDQWKKHH3HSLTA6GG4OJTUCD5IZXU7OSYVZKOXA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.891366907 (554h50m21.762449952s) [noise=true , address=145.79.227.3:4151 , piece_id=WYULE7LOQW56VSUPLSH7OAJEWG7HGX6EFPXLDCXAERK32AWBXYGQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.891881808 (554h50m21.762964853s) [noise=true , address=45.91.116.13:4155 , piece_id=QRG3MK33XHMQFYNSHOHTD7RATCWKIAFZDADKXBICH62KDFONHH5A, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.892324781 (554h50m21.763407827s) [noise=true , address=145.79.227.13:4803 , piece_id=DSIMUZ3PU4N57KCIOSQ52IX7EAKXF4OJPKH2Q57QDM3SBBVJOWWQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.893661717 (554h50m21.764744752s) [noise=true , address=145.79.227.41:4085 , piece_id=T3WB3SUSGJIK47H7EWP3WWF6GAINCTDXX3D33TDCZBL6VTFDRICQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.894123848 (554h50m21.765206893s) [noise=true , address=145.79.227.15:4873 , piece_id=Y5UZBTE2WYA7TD4H6EO4G35OXGV5PP32ST5Q4EM4RNGAUDV53TZA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.894548887 (554h50m21.765631932s) [noise=true , address=145.79.227.51:4410 , piece_id=GSQD6EJ237UUDIPY6CMGNH3SYBWQ63I66PUMYWACIS6SLQEYMGTQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.895242227 (554h50m21.766325272s) [noise=true , address=77.90.31.12:4590 , piece_id=TNQACGMQ223MB7H6EQDGQN3AFEECVTKFNYVFZ5LKNO5LPTT4HCKA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.895902985 (554h50m21.76698602s) [noise=true , address=145.79.227.19:4078 , piece_id=2KL2PML5I7QFJV5HWN3YRJSZYUQKZ53I67G4VRJBRTGJG62YXJKQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.896326401 (554h50m21.767409446s) [noise=true , address=145.79.227.26:4653 , piece_id=BQQLUK4SPHPCY5PDEEJSRBPK53BVW22MSY7V6AZ5GKE7RL2PY2JA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.896841242 (554h50m21.767924277s) [noise=true , address=45.91.116.11:4119 , piece_id=6YMLRYCO5TBOHI4ZHVSPKDIVQKHY4BGLNNMA6ZWNBT2UTQHGWB6A, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.897294275 (554h50m21.76837732s) [noise=true , address=145.79.227.33:4869 , piece_id=ENMH6OW6NBEN7IQLLIAEWD5LJBXXWUWQMOWMQNNUXVFFJ2YDC4RA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.897656704 (554h50m21.768739749s) [noise=true , address=145.79.227.39:4013 , piece_id=CZVC4FGKXWAFHGBOBUKVEEOM54ZRIVIRSFOPKSFD2RESRVS6L26A, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.898158419 (554h50m21.769241464s) [noise=true , address=77.90.31.10:4519 , piece_id=CCAK3MGMMDICKYB5WYHV446SO5S3ZMMVOJ73FICJ75FN4TLBDPWA, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.898581796 (554h50m21.769664841s) [noise=true , address=77.90.31.16:4736 , piece_id=OLS257CVX2OZ4PEIGXO5P252YTNO3HSCXKBO3ZO3COTAFF42C6AQ, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:21.898922855 (554h50m21.7700059s) [noise=true , address=145.79.227.11:4730 , piece_id=LATJLB3Z3MAPI6QPZAUYFU5NMWMKTZF3LDQPTC2MM7IULEAV6W4A, seg_pos={0 0}, upload=486438]: Uploading piece... Dec 28 18:19:23.500839465 (554h50m23.37192252s) [upload=486438]: Next returned nil segment Dec 28 18:19:23.500855124 (554h50m23.371938159s) [upload=486438]: Waiting for error group managing segments... Dec 28 18:19:23.881284027 (554h50m23.752367062s) [noise=true , address=145.79.227.19:4078 , piece_id=2KL2PML5I7QFJV5HWN3YRJSZYUQKZ53I67G4VRJBRTGJG62YXJKQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.889296886 (554h50m23.760379931s) [noise=true , address=145.79.227.41:4085 , piece_id=T3WB3SUSGJIK47H7EWP3WWF6GAINCTDXX3D33TDCZBL6VTFDRICQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.891426821 (554h50m23.762509866s) [noise=true , address=145.79.227.3:4151 , piece_id=WYULE7LOQW56VSUPLSH7OAJEWG7HGX6EFPXLDCXAERK32AWBXYGQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.896719018 (554h50m23.767802063s) [noise=true , address=145.79.227.15:4873 , piece_id=Y5UZBTE2WYA7TD4H6EO4G35OXGV5PP32ST5Q4EM4RNGAUDV53TZA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.898127370 (554h50m23.769210405s) [noise=true , address=145.79.227.26:4653 , piece_id=BQQLUK4SPHPCY5PDEEJSRBPK53BVW22MSY7V6AZ5GKE7RL2PY2JA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.901860960 (554h50m23.772944005s) [noise=true , address=145.79.227.51:4410 , piece_id=GSQD6EJ237UUDIPY6CMGNH3SYBWQ63I66PUMYWACIS6SLQEYMGTQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.910762740 (554h50m23.781845775s) [noise=true , address=145.79.227.39:4013 , piece_id=CZVC4FGKXWAFHGBOBUKVEEOM54ZRIVIRSFOPKSFD2RESRVS6L26A, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.911464987 (554h50m23.782548022s) [noise=true , address=145.79.227.8:4329 , piece_id=RBMYWNAOYFKOBNDQWKKHH3HSLTA6GG4OJTUCD5IZXU7OSYVZKOXA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.914396029 (554h50m23.785479064s) [noise=true , address=145.79.227.11:4730 , piece_id=LATJLB3Z3MAPI6QPZAUYFU5NMWMKTZF3LDQPTC2MM7IULEAV6W4A, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:23.944864977 (554h50m23.815948022s) [noise=true , address=145.79.227.13:4803 , piece_id=DSIMUZ3PU4N57KCIOSQ52IX7EAKXF4OJPKH2Q57QDM3SBBVJOWWQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:24.126100581 (554h50m23.997183626s) [noise=true , address=45.91.116.11:4119 , piece_id=6YMLRYCO5TBOHI4ZHVSPKDIVQKHY4BGLNNMA6ZWNBT2UTQHGWB6A, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:24.201780459 (554h50m24.072863504s) [noise=true , address=77.90.31.16:4736 , piece_id=OLS257CVX2OZ4PEIGXO5P252YTNO3HSCXKBO3ZO3COTAFF42C6AQ, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:24.514648785 (554h50m24.38573182s) [noise=true , address=77.90.31.10:4519 , piece_id=CCAK3MGMMDICKYB5WYHV446SO5S3ZMMVOJ73FICJ75FN4TLBDPWA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: <nil> Dec 28 18:19:24.514669073 (554h50m24.385752118s) [seg_pos={0 0}, upload=486438]: Segment reached optimal threshold of 13 pieces. Dec 28 18:19:24.514875025 (554h50m24.385958061s) [noise=true , address=145.79.227.33:4869 , piece_id=ENMH6OW6NBEN7IQLLIAEWD5LJBXXWUWQMOWMQNNUXVFFJ2YDC4RA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 18:19:24.514882840 (554h50m24.385965875s) [noise=true , address=145.79.227.33:4869 , piece_id=ENMH6OW6NBEN7IQLLIAEWD5LJBXXWUWQMOWMQNNUXVFFJ2YDC4RA, seg_pos={0 0}, upload=486438]: Long tail cancellation due to optimal threshold Dec 28 18:19:24.514993741 (554h50m24.386076796s) [noise=true , address=77.90.31.12:4590 , piece_id=TNQACGMQ223MB7H6EQDGQN3AFEECVTKFNYVFZ5LKNO5LPTT4HCKA, seg_pos={0 0}, upload=486438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 18:19:24.515004823 (554h50m24.386087868s) [noise=true , address=77.90.31.12:4590 , piece_id=TNQACGMQ223MB7H6EQDGQN3AFEECVTKFNYVFZ5LKNO5LPTT4HCKA, seg_pos={0 0}, upload=486438]: Long tail cancellation due to optimal threshold Dec 28 18:19:24.515297861 (554h50m24.386380906s) [noise=true , address=45.91.116.13:4155 , piece_id=QRG3MK33XHMQFYNSHOHTD7RATCWKIAFZDADKXBICH62KDFONHH5A, seg_pos={0 0}, upload=486438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:37338->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:37338->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: context canceled Dec 28 18:19:24.515305435 (554h50m24.38638847s) [noise=true , address=45.91.116.13:4155 , piece_id=QRG3MK33XHMQFYNSHOHTD7RATCWKIAFZDADKXBICH62KDFONHH5A, seg_pos={0 0}, upload=486438]: Long tail cancellation due to optimal threshold Dec 28 18:19:24.515321265 (554h50m24.38640431s) [seg_pos={0 0}, upload=486438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:19:24.515391148 (554h50m24.386474184s) [upload=486438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:19:24.515414183 (554h50m24.386497218s) [upload=486438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:19:24.798128851 (554h50m24.669211916s) [upload=486438]: Done uploading segments... Dec 28 18:19:24.798153016 (554h50m24.669236051s) [upload=486438]: Upload finished. err: <nil> Duration: 30.003968689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X........X.............X.X..................] duration: 18.817803756s trace: https://jaeger.dev.storj.io/trace/5eda81bca04798b7 FAIL (2025-12-28 18:19:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794293523 (554h35m21.665376568s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794729272 (554h35m21.665812307s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795130186 (554h35m21.666213231s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795534857 (554h35m21.666617902s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795928286 (554h35m21.667011331s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796356250 (554h35m21.667439285s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796770529 (554h35m21.667853574s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797234142 (554h35m21.668317187s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797701582 (554h35m21.668784697s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798145758 (554h35m21.669228793s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798717497 (554h35m21.669800532s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799245331 (554h35m21.670328377s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799828242 (554h35m21.670911287s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.800407144 (554h35m21.671490189s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.801046071 (554h35m21.672129116s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:23.074697857 (554h35m22.945780892s) [upload=486222]: Next returned nil segment Dec 28 18:04:23.074711302 (554h35m22.945794337s) [upload=486222]: Waiting for error group managing segments... Dec 28 18:04:23.417354815 (554h35m23.28843785s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.486601663 (554h35m23.357684708s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.668116249 (554h35m23.539199284s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.843876887 (554h35m23.714959922s) [noise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.848651940 (554h35m23.719734985s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.875480346 (554h35m23.746563381s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.029712621 (554h35m23.900795666s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.428381026 (554h35m24.299464061s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.550713979 (554h35m24.421797024s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.561873249 (554h35m24.432956284s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.583710861 (554h35m24.454793896s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.596115952 (554h35m24.467198997s) [noise=true , address=145.79.227.29:4959 , piece_id=F4UQHHNYPJVFEF336W3V5HIEPBKRQWM4DKOJAEXVHGULYWHLK75A, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.630336734 (554h35m24.501419779s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681392994 (554h35m24.552476029s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681422491 (554h35m24.552505556s) [seg_pos={0 0}, upload=486222]: Segment reached optimal threshold of 14 pieces. Dec 28 18:04:24.681622883 (554h35m24.552705918s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection Dec 28 18:04:24.681629966 (554h35m24.552713011s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681732601 (554h35m24.552815646s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681742510 (554h35m24.552825555s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681912925 (554h35m24.55299596s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681919507 (554h35m24.553002542s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682191224 (554h35m24.55327427s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.682198829 (554h35m24.553281864s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682218246 (554h35m24.553301281s) [seg_pos={0 0}, upload=486222]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:04:24.682292608 (554h35m24.553375653s) [upload=486222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:04:24.682297497 (554h35m24.553380532s) [upload=486222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:04:24.980627610 (554h35m24.851710655s) [upload=486222]: Done uploading segments... Dec 28 18:04:24.980654070 (554h35m24.851737105s) [upload=486222]: Upload finished. err: <nil> Duration: 18.817828944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 3.336952831s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:19:13 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.336981937s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X........X.............X.X.................] duration: 37.60795597s trace: https://jaeger.dev.storj.io/trace/5eda81bca04798b7 FAIL (2025-12-28 18:19:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794293523 (554h35m21.665376568s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.794729272 (554h35m21.665812307s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795130186 (554h35m21.666213231s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795534857 (554h35m21.666617902s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.795928286 (554h35m21.667011331s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796356250 (554h35m21.667439285s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.796770529 (554h35m21.667853574s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797234142 (554h35m21.668317187s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.797701582 (554h35m21.668784697s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798145758 (554h35m21.669228793s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.798717497 (554h35m21.669800532s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799245331 (554h35m21.670328377s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.799828242 (554h35m21.670911287s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.800407144 (554h35m21.671490189s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:21.801046071 (554h35m21.672129116s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Uploading piece... Dec 28 18:04:23.074697857 (554h35m22.945780892s) [upload=486222]: Next returned nil segment Dec 28 18:04:23.074711302 (554h35m22.945794337s) [upload=486222]: Waiting for error group managing segments... Dec 28 18:04:23.417354815 (554h35m23.28843785s) [noise=true , address=64.49.13.154:4936 , piece_id=CTNZYVAYWQYUT3RYHVIXVB6OV67JDIX6B77NRPCAS7O74VOW6HEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.486601663 (554h35m23.357684708s) [noise=true , address=64.49.13.26:4956 , piece_id=VZOKRA5SQXWZOS7K47YQQI7DKGRHA77DZM5IPK7R6LAWWW362D5Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.668116249 (554h35m23.539199284s) [noise=true , address=145.79.227.23:4568 , piece_id=DQW76R2UU2MNO3W3RZRZ4S34AW443DSSF7RTBGGNZARF4GLGA2SA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.843876887 (554h35m23.714959922s) [noise=true , address=77.90.31.19:4855 , piece_id=RTS6ZPGQ6KE3AD2LFK5LQPU652EC2JWAMJDG5VN5ZZLDDAQYJWPQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.848651940 (554h35m23.719734985s) [noise=true , address=77.90.31.17:4790 , piece_id=WQXIZ3RRWYC32PGCUZ6DBKUPTH524NDXURJALUHOIAD4QVPI55QA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:23.875480346 (554h35m23.746563381s) [noise=true , address=77.90.31.13:4641 , piece_id=MBI7UXCVYNURQXGNJCPV3I2I5BE3FOO3SVVLVHAOAUU6HBPIKMWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.029712621 (554h35m23.900795666s) [noise=true , address=77.90.31.11:4562 , piece_id=N32F26OG2NOIH24NL7WRQWNBT7UBC4WIEVG3FIH3CZ6FTH3XUMVA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.428381026 (554h35m24.299464061s) [noise=true , address=193.149.164.143:4704 , piece_id=F4NVKEG463LT32NIGTXLE7B2TIUKUW5R522MVK7Q63OJRMMW57BA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.550713979 (554h35m24.421797024s) [noise=true , address=145.79.227.44:4217 , piece_id=TIOLQOGRU7ZKBGLV3QIAITOZ7E5HEX3FU42647YF3BTIJTIROOWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.561873249 (554h35m24.432956284s) [noise=true , address=77.90.31.16:4743 , piece_id=6YQLQ72KJJWKLQDFIXQIWAPYI2LBWJL3XRXHKJYDZP3SCBUOL35Q, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.583710861 (554h35m24.454793896s) [noise=true , address=77.90.31.15:4708 , piece_id=WGPEW3ZIA2KKOCJ6RY2FCK22UBM234GELMBCFDROQRSKRZBJT5LA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.596115952 (554h35m24.467198997s) [noise=true , address=145.79.227.29:4959 , piece_id=F4UQHHNYPJVFEF336W3V5HIEPBKRQWM4DKOJAEXVHGULYWHLK75A, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.630336734 (554h35m24.501419779s) [noise=true , address=145.79.227.33:4879 , piece_id=Q4JA5DX44OWSNJMNCXW3OD3KAGVKKFJ7D6SBWUYX7QR5EQJKAPEQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681392994 (554h35m24.552476029s) [noise=true , address=145.79.227.49:4340 , piece_id=FRIICRLHNIXQHOGYHLVN6ALAO4XHDRJ67J767HJSOYGTIK6QORWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: <nil> Dec 28 18:04:24.681422491 (554h35m24.552505556s) [seg_pos={0 0}, upload=486222]: Segment reached optimal threshold of 14 pieces. Dec 28 18:04:24.681622883 (554h35m24.552705918s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39952->64.49.13.169:4715: use of closed network connection Dec 28 18:04:24.681629966 (554h35m24.552713011s) [noise=true , address=64.49.13.169:4715 , piece_id=7JJBQ2SUINZ5FJEWVW3KFNUN7ZLXVGETV4PRH36SPESLLHVCILOA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681732601 (554h35m24.552815646s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681742510 (554h35m24.552825555s) [noise=true , address=145.79.227.35:4502 , piece_id=6SYDLCYSMXITW5E4ODWTROXTUTKGPCVJGG2KAQZXPSYW72VQWDNA, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.681912925 (554h35m24.55299596s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.681919507 (554h35m24.553002542s) [noise=true , address=145.79.227.44:4210 , piece_id=F5LXQQWQ5CON6DFMBKXHMFX2OQND7YXKP77CR6NWE3ALIQQPGFWQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682191224 (554h35m24.55327427s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17rmsv4edxH6jmsN83ceiVJdfEui1qrBK5XBcf7GgvKCn2fknT): protocol: expected piece hash; context canceled; EOF Dec 28 18:04:24.682198829 (554h35m24.553281864s) [noise=true , address=145.79.227.34:4915 , piece_id=U4OIY5HO3ZDRUM62UWLCQNE7P4M37SJEQER5LOBGDPNAP7FIO2QQ, seg_pos={0 0}, upload=486222]: Long tail cancellation due to optimal threshold Dec 28 18:04:24.682218246 (554h35m24.553301281s) [seg_pos={0 0}, upload=486222]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 18:04:24.682292608 (554h35m24.553375653s) [upload=486222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:04:24.682297497 (554h35m24.553380532s) [upload=486222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:04:24.980627610 (554h35m24.851710655s) [upload=486222]: Done uploading segments... Dec 28 18:04:24.980654070 (554h35m24.851737105s) [upload=486222]: Upload finished. err: <nil> Duration: 37.607984594s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.525111008s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 18:19:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.525139371s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.205899403s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:18:39 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.205927615s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XX.X..XX........X..XX.X.XX..X....XXX..X.X....X..X..X....X.] duration: 6.044596858s trace: https://jaeger.dev.storj.io/trace/18eb52eb6a118395 FAIL (2025-12-28 18:18:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (554h48m19.069657396s) [upload=486410]: Done starting upload Dec 28 18:17:19.198639465 (554h48m19.0697225s) [upload=486410]: Uploading segments... Dec 28 18:17:19.198762329 (554h48m19.069845364s) [upload=486410]: Got next segment. Inline: false Dec 28 18:17:19.198800472 (554h48m19.069883507s) [upload=486410]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:17:19.198805211 (554h48m19.069888246s) [upload=486410]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:17:19.468130543 (554h48m19.339213588s) [seg_pos={0 0}, upload=486410]: Begin upload segment... Dec 28 18:17:19.468180258 (554h48m19.339263303s) [seg_pos={0 0}, upload=486410]: Done begin upload segment. Dec 28 18:17:19.468206268 (554h48m19.339289313s) [noise=true , address=145.79.227.36:4518 , piece_id=HIDOGWYS2B453D6DMROVFG4LDGEJ5U2O5UNKBSDRHSIQLPTO7VZQ, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.469243121 (554h48m19.340326166s) [noise=true , address=145.79.227.46:4603 , piece_id=VDG5NYWIIBYGSHGDFHHX2IMP4T6LGX37XF2OGOFFRA74TSKCVX5A, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.470030221 (554h48m19.341113266s) [noise=true , address=145.79.227.5:4223 , piece_id=3Y5SX5CZL6SUW5XKWFI426QCETDSMDNAF6OW6V6V2LN6MT2BA46Q, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.470564898 (554h48m19.341647933s) [noise=true , address=77.90.31.19:4844 , piece_id=AHWI7TBANYR6VXA73DAYCZJX5ZVBRNFDSII6PZFSNYHAHBD4JRMA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.471017510 (554h48m19.342100555s) [noise=true , address=145.79.227.7:4293 , piece_id=LUWZQJHFUOTQ2W4DT4TD7NSTK7HDPJW32NLOZTQKS4ITSQC7QXYA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.471501582 (554h48m19.342584617s) [noise=true , address=145.79.227.41:4086 , piece_id=4PVJHWHN6FW27ACIWCK62V6IZH7TMZ4RYURPEQW7QV26PUOJ2CEA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.472884315 (554h48m19.34396736s) [noise=true , address=145.79.227.45:4229 , piece_id=7UWQQTT6HEPKKIG4CWOTIDLLS7KA4REB5DGSZJDADSKS533OPPXQ, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.474346149 (554h48m19.345429194s) [noise=true , address=145.79.227.28:4727 , piece_id=VFPDDN5F3T4HADVOE3PC2VISQRFYBKLMQO2DILH6L2NDVNFV4UGQ, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.475809365 (554h48m19.34689241s) [noise=true , address=145.79.227.11:4729 , piece_id=SSTZGGDNV4FBKO57WT47ZPKBIPYIASZVODENDKGN3MMBIZ2PRKBA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.476302053 (554h48m19.347385098s) [noise=true , address=145.79.227.12:4767 , piece_id=ZSQJXNA25BPO7RSSZ2VKPVX3RAIW6TSYWVSX7GSHS27XVWFM6YWA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.477701177 (554h48m19.348784212s) [noise=true , address=77.90.31.15:4697 , piece_id=SUDLASP2YS7G4QRRI54HDL4OOAEDVKNUUIZXWD2D6YTR3XICLUBA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.479018335 (554h48m19.35010138s) [noise=true , address=77.90.31.11:4553 , piece_id=YN6YQ7FFPTFGADDHSDJRT7QFLTSABV4ZAK4NTEEUJSXHMH4VSVWQ, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.480096518 (554h48m19.351179553s) [noise=true , address=45.91.116.6:4976 , piece_id=ACPQEOEP32YETUNPHK2N2RZXIHC5KX6V2ASBUV7ECC73HDAAMUOA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.481296554 (554h48m19.352379589s) [noise=true , address=145.79.227.42:4121 , piece_id=G5SNMT2DVGVNHPEGL4CNZHJDQQKYSVP2CVDVRTXP7OX7LCYTEAWA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.481592076 (554h48m19.352675121s) [noise=true , address=145.79.227.13:4802 , piece_id=E6E2MSGO6VYETIVIGVCXSNZZCNRVHH2V2NQSBYDFK2BOZ3I6ETPA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.482931056 (554h48m19.354014101s) [noise=true , address=145.79.227.51:4411 , piece_id=AN5FXIE33V57LX3JG5BA2WH6O3DATC47G4KV4CYLE6XCDO2DPXTQ, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:21.092503974 (554h48m20.963587019s) [upload=486410]: Next returned nil segment Dec 28 18:17:21.092516157 (554h48m20.963599202s) [upload=486410]: Waiting for error group managing segments... Dec 28 18:17:21.476181453 (554h48m21.347264488s) [noise=true , address=145.79.227.36:4518 , piece_id=HIDOGWYS2B453D6DMROVFG4LDGEJ5U2O5UNKBSDRHSIQLPTO7VZQ, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.479272058 (554h48m21.350355093s) [noise=true , address=145.79.227.12:4767 , piece_id=ZSQJXNA25BPO7RSSZ2VKPVX3RAIW6TSYWVSX7GSHS27XVWFM6YWA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.479756100 (554h48m21.350839135s) [noise=true , address=145.79.227.42:4121 , piece_id=G5SNMT2DVGVNHPEGL4CNZHJDQQKYSVP2CVDVRTXP7OX7LCYTEAWA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.480152404 (554h48m21.351235439s) [noise=true , address=145.79.227.51:4411 , piece_id=AN5FXIE33V57LX3JG5BA2WH6O3DATC47G4KV4CYLE6XCDO2DPXTQ, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.482475478 (554h48m21.353558533s) [noise=true , address=145.79.227.13:4802 , piece_id=E6E2MSGO6VYETIVIGVCXSNZZCNRVHH2V2NQSBYDFK2BOZ3I6ETPA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.487444079 (554h48m21.358527114s) [noise=true , address=145.79.227.28:4727 , piece_id=VFPDDN5F3T4HADVOE3PC2VISQRFYBKLMQO2DILH6L2NDVNFV4UGQ, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.487903213 (554h48m21.358986248s) [noise=true , address=145.79.227.41:4086 , piece_id=4PVJHWHN6FW27ACIWCK62V6IZH7TMZ4RYURPEQW7QV26PUOJ2CEA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.491336461 (554h48m21.362419496s) [noise=true , address=145.79.227.45:4229 , piece_id=7UWQQTT6HEPKKIG4CWOTIDLLS7KA4REB5DGSZJDADSKS533OPPXQ, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.495586784 (554h48m21.366669829s) [noise=true , address=145.79.227.46:4603 , piece_id=VDG5NYWIIBYGSHGDFHHX2IMP4T6LGX37XF2OGOFFRA74TSKCVX5A, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.499155679 (554h48m21.370238725s) [noise=true , address=145.79.227.7:4293 , piece_id=LUWZQJHFUOTQ2W4DT4TD7NSTK7HDPJW32NLOZTQKS4ITSQC7QXYA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.499869329 (554h48m21.370952374s) [noise=true , address=145.79.227.11:4729 , piece_id=SSTZGGDNV4FBKO57WT47ZPKBIPYIASZVODENDKGN3MMBIZ2PRKBA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.688843940 (554h48m21.559926985s) [noise=true , address=77.90.31.19:4844 , piece_id=AHWI7TBANYR6VXA73DAYCZJX5ZVBRNFDSII6PZFSNYHAHBD4JRMA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.726200092 (554h48m21.597283147s) [noise=true , address=77.90.31.15:4697 , piece_id=SUDLASP2YS7G4QRRI54HDL4OOAEDVKNUUIZXWD2D6YTR3XICLUBA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.726223527 (554h48m21.597306572s) [seg_pos={0 0}, upload=486410]: Segment reached optimal threshold of 13 pieces. Dec 28 18:17:21.726663093 (554h48m21.597746148s) [noise=true , address=77.90.31.11:4553 , piece_id=YN6YQ7FFPTFGADDHSDJRT7QFLTSABV4ZAK4NTEEUJSXHMH4VSVWQ, seg_pos={0 0}, upload=486410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 18:17:21.726672522 (554h48m21.597755567s) [noise=true , address=77.90.31.11:4553 , piece_id=YN6YQ7FFPTFGADDHSDJRT7QFLTSABV4ZAK4NTEEUJSXHMH4VSVWQ, seg_pos={0 0}, upload=486410]: Long tail cancellation due to optimal threshold Dec 28 18:17:21.726842735 (554h48m21.5979258s) [noise=true , address=145.79.227.5:4223 , piece_id=3Y5SX5CZL6SUW5XKWFI426QCETDSMDNAF6OW6V6V2LN6MT2BA46Q, seg_pos={0 0}, upload=486410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:51376->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:51376->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51376->145.79.227.5:4223: use of closed network connection Dec 28 18:17:21.726862944 (554h48m21.597946009s) [noise=true , address=145.79.227.5:4223 , piece_id=3Y5SX5CZL6SUW5XKWFI426QCETDSMDNAF6OW6V6V2LN6MT2BA46Q, seg_pos={0 0}, upload=486410]: Long tail cancellation due to optimal threshold Dec 28 18:17:21.726938057 (554h48m21.598021112s) [noise=true , address=45.91.116.6:4976 , piece_id=ACPQEOEP32YETUNPHK2N2RZXIHC5KX6V2ASBUV7ECC73HDAAMUOA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:46414->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 18:17:21.726946944 (554h48m21.598029989s) [noise=true , address=45.91.116.6:4976 , piece_id=ACPQEOEP32YETUNPHK2N2RZXIHC5KX6V2ASBUV7ECC73HDAAMUOA, seg_pos={0 0}, upload=486410]: Long tail cancellation due to optimal threshold Dec 28 18:17:21.726968756 (554h48m21.598051801s) [seg_pos={0 0}, upload=486410]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:17:21.727046874 (554h48m21.598129919s) [upload=486410]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:17:21.727052345 (554h48m21.59813539s) [upload=486410]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:17:22.009060457 (554h48m21.880143502s) [upload=486410]: Done uploading segments... Dec 28 18:17:22.009088651 (554h48m21.880171686s) [upload=486410]: Upload finished. err: <nil> Duration: 6.044627127s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.489956798s trace: https://jaeger.dev.storj.io/trace/56b6bb3594a07b99 FAIL (2025-12-28 18:17:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.489985782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 3.453613572s trace: https://jaeger.dev.storj.io/trace/a4571d6e1b06393 FAIL (2025-12-28 18:17:54 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 3.453641637s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XX.X..XX........X..XX.X.XX..X....XXX..X.X....X..X..X....X] duration: 30.003497283s trace: https://jaeger.dev.storj.io/trace/18eb52eb6a118395 FAIL (2025-12-28 18:17:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (554h48m19.069657396s) [upload=486410]: Done starting upload Dec 28 18:17:19.198639465 (554h48m19.0697225s) [upload=486410]: Uploading segments... Dec 28 18:17:19.198762329 (554h48m19.069845364s) [upload=486410]: Got next segment. Inline: false Dec 28 18:17:19.198800472 (554h48m19.069883507s) [upload=486410]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 18:17:19.198805211 (554h48m19.069888246s) [upload=486410]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 18:17:19.468130543 (554h48m19.339213588s) [seg_pos={0 0}, upload=486410]: Begin upload segment... Dec 28 18:17:19.468180258 (554h48m19.339263303s) [seg_pos={0 0}, upload=486410]: Done begin upload segment. Dec 28 18:17:19.468206268 (554h48m19.339289313s) [noise=true , address=145.79.227.36:4518 , piece_id=HIDOGWYS2B453D6DMROVFG4LDGEJ5U2O5UNKBSDRHSIQLPTO7VZQ, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.469243121 (554h48m19.340326166s) [noise=true , address=145.79.227.46:4603 , piece_id=VDG5NYWIIBYGSHGDFHHX2IMP4T6LGX37XF2OGOFFRA74TSKCVX5A, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.470030221 (554h48m19.341113266s) [noise=true , address=145.79.227.5:4223 , piece_id=3Y5SX5CZL6SUW5XKWFI426QCETDSMDNAF6OW6V6V2LN6MT2BA46Q, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.470564898 (554h48m19.341647933s) [noise=true , address=77.90.31.19:4844 , piece_id=AHWI7TBANYR6VXA73DAYCZJX5ZVBRNFDSII6PZFSNYHAHBD4JRMA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.471017510 (554h48m19.342100555s) [noise=true , address=145.79.227.7:4293 , piece_id=LUWZQJHFUOTQ2W4DT4TD7NSTK7HDPJW32NLOZTQKS4ITSQC7QXYA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.471501582 (554h48m19.342584617s) [noise=true , address=145.79.227.41:4086 , piece_id=4PVJHWHN6FW27ACIWCK62V6IZH7TMZ4RYURPEQW7QV26PUOJ2CEA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.472884315 (554h48m19.34396736s) [noise=true , address=145.79.227.45:4229 , piece_id=7UWQQTT6HEPKKIG4CWOTIDLLS7KA4REB5DGSZJDADSKS533OPPXQ, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.474346149 (554h48m19.345429194s) [noise=true , address=145.79.227.28:4727 , piece_id=VFPDDN5F3T4HADVOE3PC2VISQRFYBKLMQO2DILH6L2NDVNFV4UGQ, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.475809365 (554h48m19.34689241s) [noise=true , address=145.79.227.11:4729 , piece_id=SSTZGGDNV4FBKO57WT47ZPKBIPYIASZVODENDKGN3MMBIZ2PRKBA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.476302053 (554h48m19.347385098s) [noise=true , address=145.79.227.12:4767 , piece_id=ZSQJXNA25BPO7RSSZ2VKPVX3RAIW6TSYWVSX7GSHS27XVWFM6YWA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.477701177 (554h48m19.348784212s) [noise=true , address=77.90.31.15:4697 , piece_id=SUDLASP2YS7G4QRRI54HDL4OOAEDVKNUUIZXWD2D6YTR3XICLUBA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.479018335 (554h48m19.35010138s) [noise=true , address=77.90.31.11:4553 , piece_id=YN6YQ7FFPTFGADDHSDJRT7QFLTSABV4ZAK4NTEEUJSXHMH4VSVWQ, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.480096518 (554h48m19.351179553s) [noise=true , address=45.91.116.6:4976 , piece_id=ACPQEOEP32YETUNPHK2N2RZXIHC5KX6V2ASBUV7ECC73HDAAMUOA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.481296554 (554h48m19.352379589s) [noise=true , address=145.79.227.42:4121 , piece_id=G5SNMT2DVGVNHPEGL4CNZHJDQQKYSVP2CVDVRTXP7OX7LCYTEAWA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.481592076 (554h48m19.352675121s) [noise=true , address=145.79.227.13:4802 , piece_id=E6E2MSGO6VYETIVIGVCXSNZZCNRVHH2V2NQSBYDFK2BOZ3I6ETPA, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:19.482931056 (554h48m19.354014101s) [noise=true , address=145.79.227.51:4411 , piece_id=AN5FXIE33V57LX3JG5BA2WH6O3DATC47G4KV4CYLE6XCDO2DPXTQ, seg_pos={0 0}, upload=486410]: Uploading piece... Dec 28 18:17:21.092503974 (554h48m20.963587019s) [upload=486410]: Next returned nil segment Dec 28 18:17:21.092516157 (554h48m20.963599202s) [upload=486410]: Waiting for error group managing segments... Dec 28 18:17:21.476181453 (554h48m21.347264488s) [noise=true , address=145.79.227.36:4518 , piece_id=HIDOGWYS2B453D6DMROVFG4LDGEJ5U2O5UNKBSDRHSIQLPTO7VZQ, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.479272058 (554h48m21.350355093s) [noise=true , address=145.79.227.12:4767 , piece_id=ZSQJXNA25BPO7RSSZ2VKPVX3RAIW6TSYWVSX7GSHS27XVWFM6YWA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.479756100 (554h48m21.350839135s) [noise=true , address=145.79.227.42:4121 , piece_id=G5SNMT2DVGVNHPEGL4CNZHJDQQKYSVP2CVDVRTXP7OX7LCYTEAWA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.480152404 (554h48m21.351235439s) [noise=true , address=145.79.227.51:4411 , piece_id=AN5FXIE33V57LX3JG5BA2WH6O3DATC47G4KV4CYLE6XCDO2DPXTQ, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.482475478 (554h48m21.353558533s) [noise=true , address=145.79.227.13:4802 , piece_id=E6E2MSGO6VYETIVIGVCXSNZZCNRVHH2V2NQSBYDFK2BOZ3I6ETPA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.487444079 (554h48m21.358527114s) [noise=true , address=145.79.227.28:4727 , piece_id=VFPDDN5F3T4HADVOE3PC2VISQRFYBKLMQO2DILH6L2NDVNFV4UGQ, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.487903213 (554h48m21.358986248s) [noise=true , address=145.79.227.41:4086 , piece_id=4PVJHWHN6FW27ACIWCK62V6IZH7TMZ4RYURPEQW7QV26PUOJ2CEA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.491336461 (554h48m21.362419496s) [noise=true , address=145.79.227.45:4229 , piece_id=7UWQQTT6HEPKKIG4CWOTIDLLS7KA4REB5DGSZJDADSKS533OPPXQ, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.495586784 (554h48m21.366669829s) [noise=true , address=145.79.227.46:4603 , piece_id=VDG5NYWIIBYGSHGDFHHX2IMP4T6LGX37XF2OGOFFRA74TSKCVX5A, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.499155679 (554h48m21.370238725s) [noise=true , address=145.79.227.7:4293 , piece_id=LUWZQJHFUOTQ2W4DT4TD7NSTK7HDPJW32NLOZTQKS4ITSQC7QXYA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.499869329 (554h48m21.370952374s) [noise=true , address=145.79.227.11:4729 , piece_id=SSTZGGDNV4FBKO57WT47ZPKBIPYIASZVODENDKGN3MMBIZ2PRKBA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.688843940 (554h48m21.559926985s) [noise=true , address=77.90.31.19:4844 , piece_id=AHWI7TBANYR6VXA73DAYCZJX5ZVBRNFDSII6PZFSNYHAHBD4JRMA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.726200092 (554h48m21.597283147s) [noise=true , address=77.90.31.15:4697 , piece_id=SUDLASP2YS7G4QRRI54HDL4OOAEDVKNUUIZXWD2D6YTR3XICLUBA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: <nil> Dec 28 18:17:21.726223527 (554h48m21.597306572s) [seg_pos={0 0}, upload=486410]: Segment reached optimal threshold of 13 pieces. Dec 28 18:17:21.726663093 (554h48m21.597746148s) [noise=true , address=77.90.31.11:4553 , piece_id=YN6YQ7FFPTFGADDHSDJRT7QFLTSABV4ZAK4NTEEUJSXHMH4VSVWQ, seg_pos={0 0}, upload=486410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 28 18:17:21.726672522 (554h48m21.597755567s) [noise=true , address=77.90.31.11:4553 , piece_id=YN6YQ7FFPTFGADDHSDJRT7QFLTSABV4ZAK4NTEEUJSXHMH4VSVWQ, seg_pos={0 0}, upload=486410]: Long tail cancellation due to optimal threshold Dec 28 18:17:21.726842735 (554h48m21.5979258s) [noise=true , address=145.79.227.5:4223 , piece_id=3Y5SX5CZL6SUW5XKWFI426QCETDSMDNAF6OW6V6V2LN6MT2BA46Q, seg_pos={0 0}, upload=486410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:51376->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:51376->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51376->145.79.227.5:4223: use of closed network connection Dec 28 18:17:21.726862944 (554h48m21.597946009s) [noise=true , address=145.79.227.5:4223 , piece_id=3Y5SX5CZL6SUW5XKWFI426QCETDSMDNAF6OW6V6V2LN6MT2BA46Q, seg_pos={0 0}, upload=486410]: Long tail cancellation due to optimal threshold Dec 28 18:17:21.726938057 (554h48m21.598021112s) [noise=true , address=45.91.116.6:4976 , piece_id=ACPQEOEP32YETUNPHK2N2RZXIHC5KX6V2ASBUV7ECC73HDAAMUOA, seg_pos={0 0}, upload=486410]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: read tcp 188.166.225.170:46414->45.91.116.6:4976: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 18:17:21.726946944 (554h48m21.598029989s) [noise=true , address=45.91.116.6:4976 , piece_id=ACPQEOEP32YETUNPHK2N2RZXIHC5KX6V2ASBUV7ECC73HDAAMUOA, seg_pos={0 0}, upload=486410]: Long tail cancellation due to optimal threshold Dec 28 18:17:21.726968756 (554h48m21.598051801s) [seg_pos={0 0}, upload=486410]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 18:17:21.727046874 (554h48m21.598129919s) [upload=486410]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 18:17:21.727052345 (554h48m21.59813539s) [upload=486410]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 18:17:22.009060457 (554h48m21.880143502s) [upload=486410]: Done uploading segments... Dec 28 18:17:22.009088651 (554h48m21.880171686s) [upload=486410]: Upload finished. err: <nil> Duration: 30.003521329s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)