Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....XX] duration: 30.003996365s trace: https://jaeger.dev.storj.io/trace/aa8c405b9d6f89a FAIL (2025-12-28 05:03:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 05:03:25.083539577 (541h34m24.954622622s) [upload=474930]: Starting upload Dec 28 05:03:25.083626612 (541h34m24.954709647s) [upload=474930]: Done starting upload Dec 28 05:03:25.083696525 (541h34m24.95477957s) [upload=474930]: Uploading segments... Dec 28 05:03:25.083872230 (541h34m24.954955275s) [upload=474930]: Got next segment. Inline: false Dec 28 05:03:25.083929319 (541h34m24.955012364s) [upload=474930]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:03:25.083936141 (541h34m24.955019176s) [upload=474930]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:03:25.338498336 (541h34m25.209581371s) [seg_pos={0 0}, upload=474930]: Begin upload segment... Dec 28 05:03:25.338549704 (541h34m25.209632739s) [seg_pos={0 0}, upload=474930]: Done begin upload segment. Dec 28 05:03:25.338577747 (541h34m25.209660792s) [noise=true , address=145.79.227.32:4835 , piece_id=MNUQQ3RJFOLABIFONWGC2OBOPZO4A2UJYI672XJCNKIYJF66P4FA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.338833044 (541h34m25.209916089s) [noise=true , address=145.79.227.39:4015 , piece_id=RBTTQQPYQSK7SLGV5SJEXCCU3HCCWRWLC227ZE3U3X2PWX37MWFA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.339414630 (541h34m25.210497666s) [noise=true , address=45.91.116.13:4156 , piece_id=TEOU6N5G6AP4LORNRMLALA477I7RCSZY6F3DITCAR5IEZUQTO4LA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.339766912 (541h34m25.210849947s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340057866 (541h34m25.211140911s) [noise=true , address=145.79.227.48:4301 , piece_id=YR4QXH66TXMJV5SGSIOH4UMBWA3P2YTLJB6H2MGMNK5OCX3JBPZA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340407872 (541h34m25.211490907s) [noise=true , address=45.91.116.7:4009 , piece_id=RR2TRERI5SMJKO7GBOQTFMBL5MX24KBFMSMSKDKJBUZ25HSPUP3Q, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.340782987 (541h34m25.211866032s) [noise=true , address=145.79.227.51:4411 , piece_id=CFODJTDCC6TY2PD6EC7HQSVC46TYHN7GAZV5YJCI5YRIWT5X3BVQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.341118505 (541h34m25.21220155s) [noise=true , address=145.79.227.42:4121 , piece_id=IO7SWIKWSKSE74SMXMYF7LO2DWKJ3Z5NGK5NJM544V6ACXEGH4BQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.341480525 (541h34m25.21256356s) [noise=true , address=145.79.227.50:4374 , piece_id=53LJ3CVTG3K7KHH2ISTTPXFFPZ6J3DWBDSG5ENR5ID3WSLHXKJBA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.342699426 (541h34m25.213782471s) [noise=true , address=45.91.116.9:4261 , piece_id=M5MMGKBOCO3N2KQI4ZUC2CHO63O2HY4RX7A5OZ54N4ZDCBVAB52Q, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.342741726 (541h34m25.213824761s) [noise=true , address=77.90.31.11:4555 , piece_id=RPMB4SBOZADVIX26S7354XVJX2IVMJ23CFOTJEZFDRPQIWJQNTXQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.343815331 (541h34m25.214898376s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.343923096 (541h34m25.215006131s) [noise=true , address=145.79.227.41:4085 , piece_id=GEPDJ4TLJMLWPUXTZK4GR23AUDI24NWGUCRWCT77KKWTPZPZVHQA, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.344923541 (541h34m25.216006586s) [noise=true , address=145.79.227.17:4970 , piece_id=NQEJ5AKQJW3JJ2P7Z7OI23F4256A4F2ATD2DOBPEGO5ZM45LIYTQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.344997201 (541h34m25.216080236s) [noise=true , address=145.79.227.47:4267 , piece_id=VLDUJO2SZET4JEYSK3PQDPMLV7N3NFUK3SWAPLYGMAV73V4MRX4A, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:25.346392789 (541h34m25.217475824s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Uploading piece... Dec 28 05:03:26.952520197 (541h34m26.823603242s) [upload=474930]: Next returned nil segment Dec 28 05:03:26.952536228 (541h34m26.823619273s) [upload=474930]: Waiting for error group managing segments... Dec 28 05:03:27.339291868 (541h34m27.210374913s) [noise=true , address=145.79.227.48:4301 , piece_id=YR4QXH66TXMJV5SGSIOH4UMBWA3P2YTLJB6H2MGMNK5OCX3JBPZA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.339967505 (541h34m27.21105054s) [noise=true , address=145.79.227.32:4835 , piece_id=MNUQQ3RJFOLABIFONWGC2OBOPZO4A2UJYI672XJCNKIYJF66P4FA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.351587261 (541h34m27.222670306s) [noise=true , address=145.79.227.47:4267 , piece_id=VLDUJO2SZET4JEYSK3PQDPMLV7N3NFUK3SWAPLYGMAV73V4MRX4A, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.351742316 (541h34m27.222825351s) [noise=true , address=145.79.227.42:4121 , piece_id=IO7SWIKWSKSE74SMXMYF7LO2DWKJ3Z5NGK5NJM544V6ACXEGH4BQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.352351776 (541h34m27.223434821s) [noise=true , address=145.79.227.41:4085 , piece_id=GEPDJ4TLJMLWPUXTZK4GR23AUDI24NWGUCRWCT77KKWTPZPZVHQA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.352757619 (541h34m27.223840664s) [noise=true , address=145.79.227.17:4970 , piece_id=NQEJ5AKQJW3JJ2P7Z7OI23F4256A4F2ATD2DOBPEGO5ZM45LIYTQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.357654343 (541h34m27.228737378s) [noise=true , address=145.79.227.51:4411 , piece_id=CFODJTDCC6TY2PD6EC7HQSVC46TYHN7GAZV5YJCI5YRIWT5X3BVQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.362524657 (541h34m27.233607702s) [noise=true , address=145.79.227.39:4015 , piece_id=RBTTQQPYQSK7SLGV5SJEXCCU3HCCWRWLC227ZE3U3X2PWX37MWFA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.391519668 (541h34m27.262602704s) [noise=true , address=145.79.227.50:4374 , piece_id=53LJ3CVTG3K7KHH2ISTTPXFFPZ6J3DWBDSG5ENR5ID3WSLHXKJBA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.586655226 (541h34m27.457738271s) [noise=true , address=45.91.116.7:4009 , piece_id=RR2TRERI5SMJKO7GBOQTFMBL5MX24KBFMSMSKDKJBUZ25HSPUP3Q, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.615530559 (541h34m27.486613594s) [noise=true , address=77.90.31.11:4555 , piece_id=RPMB4SBOZADVIX26S7354XVJX2IVMJ23CFOTJEZFDRPQIWJQNTXQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.675202583 (541h34m27.546285629s) [noise=true , address=45.91.116.9:4261 , piece_id=M5MMGKBOCO3N2KQI4ZUC2CHO63O2HY4RX7A5OZ54N4ZDCBVAB52Q, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.997707050 (541h34m27.868790095s) [noise=true , address=45.91.116.13:4156 , piece_id=TEOU6N5G6AP4LORNRMLALA477I7RCSZY6F3DITCAR5IEZUQTO4LA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: <nil> Dec 28 05:03:27.997732028 (541h34m27.868815063s) [seg_pos={0 0}, upload=474930]: Segment reached optimal threshold of 13 pieces. Dec 28 05:03:27.997981663 (541h34m27.869064708s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF Dec 28 05:03:27.998040785 (541h34m27.86912384s) [noise=true , address=77.90.31.19:4844 , piece_id=LOG3JZ3PDYAI2IQP5KWZJXETNTLIVZ4F5RTVQC5HYJ5EOZRSJIPA, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998200149 (541h34m27.869283194s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 28 05:03:27.998211119 (541h34m27.869294175s) [noise=true , address=145.79.227.30:4761 , piece_id=OBY5HHB6IR3YNUT7CS4TPYCXJSKIC47GUNMKU7TGM26RTS7I2IZQ, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998223263 (541h34m27.869306298s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51532->77.90.31.17:4770: use of closed network connection Dec 28 05:03:27.998230527 (541h34m27.869313592s) [noise=true , address=77.90.31.17:4770 , piece_id=BDPPRSZDXGSVMUNZDLNV6ZAJCJEVXR2OSYAE333G2EZULC45HUCQ, seg_pos={0 0}, upload=474930]: Long tail cancellation due to optimal threshold Dec 28 05:03:27.998248771 (541h34m27.869331816s) [seg_pos={0 0}, upload=474930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:03:27.998331740 (541h34m27.869414785s) [upload=474930]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:03:27.998337981 (541h34m27.869421026s) [upload=474930]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:03:28.348380875 (541h34m28.21946394s) [upload=474930]: Done uploading segments... Dec 28 05:03:28.348406634 (541h34m28.219489669s) [upload=474930]: Upload finished. err: <nil> Duration: 30.004022165s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..X] duration: 5.001978554s trace: https://jaeger.dev.storj.io/trace/9f81e4fe0da6893 FAIL (2025-12-28 05:03:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002004494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.397073303s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:03:40 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 6.397102938s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X................................................X.........] duration: 45.522240077s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:03:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 45.522267489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.940246053s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:03:17 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.940285479s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....X] duration: 30.002954931s trace: https://jaeger.dev.storj.io/trace/20d5c6af2296cbf0 FAIL (2025-12-28 05:02:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: YTWAVPDPZKBH4AE2TOUJ7ZQCACAJH2COG3JLZ2VHOKPXEYZGKFYA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: 7TKHCNLFATST4VPWMWKZG5MJHQYBSEJ2UOXTWZAWQTVDEIYVT4SA) context deadline exceeded ======LOGS====== 8 05:02:26.828558837 (541h33m26.699641872s) [upload=474914]: Starting upload Dec 28 05:02:26.828622699 (541h33m26.699705734s) [upload=474914]: Done starting upload Dec 28 05:02:26.828825886 (541h33m26.699908931s) [upload=474914]: Uploading segments... Dec 28 05:02:26.828886511 (541h33m26.699969546s) [upload=474914]: Got next segment. Inline: false Dec 28 05:02:26.828918031 (541h33m26.700001066s) [upload=474914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 05:02:26.828922981 (541h33m26.700006016s) [upload=474914]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 05:02:27.079426667 (541h33m26.950509712s) [seg_pos={0 0}, upload=474914]: Begin upload segment... Dec 28 05:02:27.079499807 (541h33m26.950582842s) [seg_pos={0 0}, upload=474914]: Done begin upload segment. Dec 28 05:02:27.079538009 (541h33m26.950621044s) [noise=true , address=145.79.227.52:4445 , piece_id=5QCSQ7NN3SL3NMVI4AK4KWJIXCCL5UOSIW64E3JKKH53MOPTMQNQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.079638531 (541h33m26.950721576s) [noise=true , address=145.79.227.7:4293 , piece_id=HYHCYV5IAWJYEMFE57OVAAOPXF3Z6ZZ6RVS5JFSQIXDKZ2DEU2GQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.080279702 (541h33m26.951362747s) [noise=true , address=45.91.116.11:4118 , piece_id=COCMPWPVJ6GJN3HI5SMZ6YGFPI4N5QVE6VJ7MYREFVEECUJMKRUQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.080625741 (541h33m26.951708776s) [noise=true , address=145.79.227.14:4839 , piece_id=NA7CGW3LBYCKUE5DX6HJPDLYWQIMDKMXOKRYOTHX5AQKRDYADYQA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.081165969 (541h33m26.952249014s) [noise=true , address=145.79.227.51:4411 , piece_id=SAHB5CSKTHUTEAPJSVKNYCTLAUMUP3N3HBXUZ746KX6X6TMBD4SQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.081328389 (541h33m26.952411424s) [noise=true , address=145.79.227.31:4799 , piece_id=5NOMGHHQN3T6YYQ3R2Y7K3BKITIBOSNB5CBEXCEQIYSTHTTZSKRQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.081555922 (541h33m26.952638967s) [noise=true , address=145.79.227.25:4617 , piece_id=YFBYT47H7WLRQGYTZXX7GKKZLYSURZMHIM2BUZBZ22W32TZAC52Q, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.081856274 (541h33m26.952939309s) [noise=true , address=77.90.31.10:4519 , piece_id=R6WT6MTC7H3IVUO2X3UB5COMENHHN34NAQCEAW2VCDDDQS7U2DWQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.082005778 (541h33m26.953088823s) [noise=true , address=145.79.227.9:4365 , piece_id=YO3O23PTGMLNQKSXTR6UNIHTIDMXJ2CNTTWD7YM5I4MKLADZKGEA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.082262498 (541h33m26.953345543s) [noise=true , address=145.79.227.12:4767 , piece_id=RT6G4JVCURUQMVQLCQWMBCQRJQ4VJ2WY4W4BUEMXHVIMUXQDHBUQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.082496813 (541h33m26.953579858s) [noise=true , address=77.90.31.12:4590 , piece_id=YTWAVPDPZKBH4AE2TOUJ7ZQCACAJH2COG3JLZ2VHOKPXEYZGKFYA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.082802195 (541h33m26.95388523s) [noise=true , address=145.79.227.3:4150 , piece_id=PHOQPLSF6K3DHBXB3UWZ54PJCWYLHSEVBQNRKMSBY6HH24GFDDQA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.083124458 (541h33m26.954207503s) [noise=true , address=45.91.116.13:4156 , piece_id=7TKHCNLFATST4VPWMWKZG5MJHQYBSEJ2UOXTWZAWQTVDEIYVT4SA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.083202047 (541h33m26.954285092s) [noise=true , address=45.91.116.8:4045 , piece_id=TEZFSZWTMK5ONII4QDWSW752F3MOC675NEOW6N3JYB42JYTZZQJA, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.083570448 (541h33m26.954653493s) [noise=true , address=45.91.116.10:4082 , piece_id=F4ZUIYQ6LB3Z5CYVWENMUUREVHV26JYFGQU6UTBKYMOV7SBJYTFQ, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:27.083618188 (541h33m26.954701223s) [noise=true , address=145.79.227.17:4971 , piece_id=QLYXCX3VE4H46YM7DFJLQ7WQFRPI7XBPIVL2PZPQPKBZVQPQRL2Q, seg_pos={0 0}, upload=474914]: Uploading piece... Dec 28 05:02:28.719685608 (541h33m28.590768653s) [upload=474914]: Next returned nil segment Dec 28 05:02:28.719706087 (541h33m28.590789132s) [upload=474914]: Waiting for error group managing segments... Dec 28 05:02:29.108045794 (541h33m28.979128829s) [noise=true , address=145.79.227.17:4971 , piece_id=QLYXCX3VE4H46YM7DFJLQ7WQFRPI7XBPIVL2PZPQPKBZVQPQRL2Q, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.109514390 (541h33m28.980597425s) [noise=true , address=145.79.227.14:4839 , piece_id=NA7CGW3LBYCKUE5DX6HJPDLYWQIMDKMXOKRYOTHX5AQKRDYADYQA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.119441714 (541h33m28.990524759s) [noise=true , address=145.79.227.52:4445 , piece_id=5QCSQ7NN3SL3NMVI4AK4KWJIXCCL5UOSIW64E3JKKH53MOPTMQNQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.120192343 (541h33m28.991275378s) [noise=true , address=145.79.227.3:4150 , piece_id=PHOQPLSF6K3DHBXB3UWZ54PJCWYLHSEVBQNRKMSBY6HH24GFDDQA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.120851689 (541h33m28.991934734s) [noise=true , address=145.79.227.7:4293 , piece_id=HYHCYV5IAWJYEMFE57OVAAOPXF3Z6ZZ6RVS5JFSQIXDKZ2DEU2GQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.121343044 (541h33m28.992426079s) [noise=true , address=145.79.227.31:4799 , piece_id=5NOMGHHQN3T6YYQ3R2Y7K3BKITIBOSNB5CBEXCEQIYSTHTTZSKRQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.125167687 (541h33m28.996250722s) [noise=true , address=145.79.227.51:4411 , piece_id=SAHB5CSKTHUTEAPJSVKNYCTLAUMUP3N3HBXUZ746KX6X6TMBD4SQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.145068240 (541h33m29.016151275s) [noise=true , address=145.79.227.25:4617 , piece_id=YFBYT47H7WLRQGYTZXX7GKKZLYSURZMHIM2BUZBZ22W32TZAC52Q, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.154369251 (541h33m29.025452286s) [noise=true , address=145.79.227.12:4767 , piece_id=RT6G4JVCURUQMVQLCQWMBCQRJQ4VJ2WY4W4BUEMXHVIMUXQDHBUQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.242443788 (541h33m29.113526843s) [noise=true , address=145.79.227.9:4365 , piece_id=YO3O23PTGMLNQKSXTR6UNIHTIDMXJ2CNTTWD7YM5I4MKLADZKGEA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.356424034 (541h33m29.227507069s) [noise=true , address=77.90.31.12:4590 , piece_id=YTWAVPDPZKBH4AE2TOUJ7ZQCACAJH2COG3JLZ2VHOKPXEYZGKFYA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.474876244 (541h33m29.34595928s) [noise=true , address=45.91.116.13:4156 , piece_id=7TKHCNLFATST4VPWMWKZG5MJHQYBSEJ2UOXTWZAWQTVDEIYVT4SA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.640545333 (541h33m29.511628368s) [noise=true , address=45.91.116.11:4118 , piece_id=COCMPWPVJ6GJN3HI5SMZ6YGFPI4N5QVE6VJ7MYREFVEECUJMKRUQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: <nil> Dec 28 05:02:29.640563889 (541h33m29.511646924s) [seg_pos={0 0}, upload=474914]: Segment reached optimal threshold of 13 pieces. Dec 28 05:02:29.640723252 (541h33m29.511806287s) [noise=true , address=77.90.31.10:4519 , piece_id=R6WT6MTC7H3IVUO2X3UB5COMENHHN34NAQCEAW2VCDDDQS7U2DWQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: write tcp 188.166.225.170:60168->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:60168->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60168->77.90.31.10:4519: use of closed network connection Dec 28 05:02:29.640729453 (541h33m29.511812498s) [noise=true , address=77.90.31.10:4519 , piece_id=R6WT6MTC7H3IVUO2X3UB5COMENHHN34NAQCEAW2VCDDDQS7U2DWQ, seg_pos={0 0}, upload=474914]: Long tail cancellation due to optimal threshold Dec 28 05:02:29.640938251 (541h33m29.512021296s) [noise=true , address=45.91.116.10:4082 , piece_id=F4ZUIYQ6LB3Z5CYVWENMUUREVHV26JYFGQU6UTBKYMOV7SBJYTFQ, seg_pos={0 0}, upload=474914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 05:02:29.640949353 (541h33m29.512032398s) [noise=true , address=45.91.116.10:4082 , piece_id=F4ZUIYQ6LB3Z5CYVWENMUUREVHV26JYFGQU6UTBKYMOV7SBJYTFQ, seg_pos={0 0}, upload=474914]: Long tail cancellation due to optimal threshold Dec 28 05:02:29.641046196 (541h33m29.512129231s) [noise=true , address=45.91.116.8:4045 , piece_id=TEZFSZWTMK5ONII4QDWSW752F3MOC675NEOW6N3JYB42JYTZZQJA, seg_pos={0 0}, upload=474914]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 05:02:29.641050966 (541h33m29.512134001s) [noise=true , address=45.91.116.8:4045 , piece_id=TEZFSZWTMK5ONII4QDWSW752F3MOC675NEOW6N3JYB42JYTZZQJA, seg_pos={0 0}, upload=474914]: Long tail cancellation due to optimal threshold Dec 28 05:02:29.641063941 (541h33m29.512146986s) [seg_pos={0 0}, upload=474914]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 05:02:29.641150074 (541h33m29.512233109s) [upload=474914]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 05:02:29.641154634 (541h33m29.512237679s) [upload=474914]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 05:02:29.947730710 (541h33m29.818813766s) [upload=474914]: Done uploading segments... Dec 28 05:02:29.947756619 (541h33m29.818839674s) [upload=474914]: Upload finished. err: <nil> Duration: 30.003003854s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X................................................X........] duration: 28.835529334s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:02:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 28.835559751s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX..] duration: 4.36848358s trace: https://jaeger.dev.storj.io/trace/7d10056725899236 FAIL (2025-12-28 05:02:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.368510171s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.238455988s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:02:15 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.238481966s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X................................................X.......] duration: 55.17438591s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:02:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 55.174431036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.....] duration: 6.043356187s trace: https://jaeger.dev.storj.io/trace/4cd911d6d56e2fc8 FAIL (2025-12-28 05:02:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A) context deadline exceeded ======LOGS====== Dec 28 04:57:28.451800538 (541h28m28.322883604s) [upload=474844]: Starting upload Dec 28 04:57:28.451928753 (541h28m28.323011818s) [upload=474844]: Done starting upload Dec 28 04:57:28.451995079 (541h28m28.323078114s) [upload=474844]: Uploading segments... Dec 28 04:57:28.452185842 (541h28m28.323268887s) [upload=474844]: Got next segment. Inline: false Dec 28 04:57:28.452234825 (541h28m28.32331787s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:57:28.452240607 (541h28m28.323323642s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:57:28.695520939 (541h28m28.566603974s) [seg_pos={0 0}, upload=474844]: Begin upload segment... Dec 28 04:57:28.695571014 (541h28m28.566654049s) [seg_pos={0 0}, upload=474844]: Done begin upload segment. Dec 28 04:57:28.695588708 (541h28m28.566671753s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.695600470 (541h28m28.566683505s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696371448 (541h28m28.567454493s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696796467 (541h28m28.567879502s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.697682154 (541h28m28.568765199s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.698021711 (541h28m28.569104746s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699043126 (541h28m28.570126161s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699080727 (541h28m28.570163772s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700036667 (541h28m28.571119702s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700097854 (541h28m28.571180889s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.701122475 (541h28m28.57220552s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.702295218 (541h28m28.573378253s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.703764426 (541h28m28.574847471s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.704742959 (541h28m28.575826004s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.705869163 (541h28m28.576952208s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.706067191 (541h28m28.577150226s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:30.344383692 (541h28m30.215466727s) [upload=474844]: Next returned nil segment Dec 28 04:57:30.344408439 (541h28m30.215491494s) [upload=474844]: Waiting for error group managing segments... Dec 28 04:57:30.705618958 (541h28m30.576701993s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.710373342 (541h28m30.581456387s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.717849567 (541h28m30.588932612s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731461376 (541h28m30.602544421s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731614708 (541h28m30.602697753s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.737546954 (541h28m30.608629989s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748258019 (541h28m30.619341064s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748985264 (541h28m30.620068309s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.775426382 (541h28m30.646509427s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.957930196 (541h28m30.829013241s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.996359071 (541h28m30.867442116s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.061702074 (541h28m30.932785119s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091674256 (541h28m30.962757301s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091697750 (541h28m30.962780795s) [seg_pos={0 0}, upload=474844]: Segment reached optimal threshold of 13 pieces. Dec 28 04:57:31.091956844 (541h28m30.963039889s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.091967544 (541h28m30.963050589s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.091992962 (541h28m30.963075998s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092001779 (541h28m30.963084824s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092171182 (541h28m30.963254227s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092177233 (541h28m30.963260278s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092200828 (541h28m30.963283863s) [seg_pos={0 0}, upload=474844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:57:31.092277013 (541h28m30.963360058s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:57:31.092282724 (541h28m30.963365769s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:57:31.376618933 (541h28m31.247701978s) [upload=474844]: Done uploading segments... Dec 28 04:57:31.376642698 (541h28m31.247725743s) [upload=474844]: Upload finished. err: <nil> Duration: 6.043390995s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [XXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX.] duration: 5.000815676s trace: https://jaeger.dev.storj.io/trace/7d10056725899236 FAIL (2025-12-28 05:02:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000842267s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXXX] duration: 5.001491448s trace: https://jaeger.dev.storj.io/trace/7d10056725899236 FAIL (2025-12-28 05:01:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001513911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX....] duration: 6.196064023s trace: https://jaeger.dev.storj.io/trace/4cd911d6d56e2fc8 FAIL (2025-12-28 05:01:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A) context deadline exceeded ======LOGS====== Dec 28 04:57:28.451800538 (541h28m28.322883604s) [upload=474844]: Starting upload Dec 28 04:57:28.451928753 (541h28m28.323011818s) [upload=474844]: Done starting upload Dec 28 04:57:28.451995079 (541h28m28.323078114s) [upload=474844]: Uploading segments... Dec 28 04:57:28.452185842 (541h28m28.323268887s) [upload=474844]: Got next segment. Inline: false Dec 28 04:57:28.452234825 (541h28m28.32331787s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:57:28.452240607 (541h28m28.323323642s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:57:28.695520939 (541h28m28.566603974s) [seg_pos={0 0}, upload=474844]: Begin upload segment... Dec 28 04:57:28.695571014 (541h28m28.566654049s) [seg_pos={0 0}, upload=474844]: Done begin upload segment. Dec 28 04:57:28.695588708 (541h28m28.566671753s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.695600470 (541h28m28.566683505s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696371448 (541h28m28.567454493s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696796467 (541h28m28.567879502s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.697682154 (541h28m28.568765199s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.698021711 (541h28m28.569104746s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699043126 (541h28m28.570126161s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699080727 (541h28m28.570163772s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700036667 (541h28m28.571119702s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700097854 (541h28m28.571180889s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.701122475 (541h28m28.57220552s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.702295218 (541h28m28.573378253s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.703764426 (541h28m28.574847471s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.704742959 (541h28m28.575826004s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.705869163 (541h28m28.576952208s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.706067191 (541h28m28.577150226s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:30.344383692 (541h28m30.215466727s) [upload=474844]: Next returned nil segment Dec 28 04:57:30.344408439 (541h28m30.215491494s) [upload=474844]: Waiting for error group managing segments... Dec 28 04:57:30.705618958 (541h28m30.576701993s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.710373342 (541h28m30.581456387s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.717849567 (541h28m30.588932612s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731461376 (541h28m30.602544421s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731614708 (541h28m30.602697753s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.737546954 (541h28m30.608629989s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748258019 (541h28m30.619341064s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748985264 (541h28m30.620068309s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.775426382 (541h28m30.646509427s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.957930196 (541h28m30.829013241s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.996359071 (541h28m30.867442116s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.061702074 (541h28m30.932785119s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091674256 (541h28m30.962757301s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091697750 (541h28m30.962780795s) [seg_pos={0 0}, upload=474844]: Segment reached optimal threshold of 13 pieces. Dec 28 04:57:31.091956844 (541h28m30.963039889s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.091967544 (541h28m30.963050589s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.091992962 (541h28m30.963075998s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092001779 (541h28m30.963084824s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092171182 (541h28m30.963254227s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092177233 (541h28m30.963260278s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092200828 (541h28m30.963283863s) [seg_pos={0 0}, upload=474844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:57:31.092277013 (541h28m30.963360058s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:57:31.092282724 (541h28m30.963365769s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:57:31.376618933 (541h28m31.247701978s) [upload=474844]: Done uploading segments... Dec 28 04:57:31.376642698 (541h28m31.247725743s) [upload=474844]: Upload finished. err: <nil> Duration: 6.196092638s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.335210413s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:01:11 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.335236142s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [.XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXXX] duration: 5.001672427s trace: https://jaeger.dev.storj.io/trace/3a730a56361c682a FAIL (2025-12-28 05:01:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001696634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X................................................X......] duration: 36.274527755s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:00:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 36.274556008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX...] duration: 6.128131525s trace: https://jaeger.dev.storj.io/trace/4cd911d6d56e2fc8 FAIL (2025-12-28 05:00:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A) context deadline exceeded ======LOGS====== Dec 28 04:57:28.451800538 (541h28m28.322883604s) [upload=474844]: Starting upload Dec 28 04:57:28.451928753 (541h28m28.323011818s) [upload=474844]: Done starting upload Dec 28 04:57:28.451995079 (541h28m28.323078114s) [upload=474844]: Uploading segments... Dec 28 04:57:28.452185842 (541h28m28.323268887s) [upload=474844]: Got next segment. Inline: false Dec 28 04:57:28.452234825 (541h28m28.32331787s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:57:28.452240607 (541h28m28.323323642s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:57:28.695520939 (541h28m28.566603974s) [seg_pos={0 0}, upload=474844]: Begin upload segment... Dec 28 04:57:28.695571014 (541h28m28.566654049s) [seg_pos={0 0}, upload=474844]: Done begin upload segment. Dec 28 04:57:28.695588708 (541h28m28.566671753s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.695600470 (541h28m28.566683505s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696371448 (541h28m28.567454493s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696796467 (541h28m28.567879502s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.697682154 (541h28m28.568765199s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.698021711 (541h28m28.569104746s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699043126 (541h28m28.570126161s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699080727 (541h28m28.570163772s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700036667 (541h28m28.571119702s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700097854 (541h28m28.571180889s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.701122475 (541h28m28.57220552s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.702295218 (541h28m28.573378253s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.703764426 (541h28m28.574847471s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.704742959 (541h28m28.575826004s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.705869163 (541h28m28.576952208s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.706067191 (541h28m28.577150226s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:30.344383692 (541h28m30.215466727s) [upload=474844]: Next returned nil segment Dec 28 04:57:30.344408439 (541h28m30.215491494s) [upload=474844]: Waiting for error group managing segments... Dec 28 04:57:30.705618958 (541h28m30.576701993s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.710373342 (541h28m30.581456387s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.717849567 (541h28m30.588932612s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731461376 (541h28m30.602544421s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731614708 (541h28m30.602697753s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.737546954 (541h28m30.608629989s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748258019 (541h28m30.619341064s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748985264 (541h28m30.620068309s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.775426382 (541h28m30.646509427s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.957930196 (541h28m30.829013241s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.996359071 (541h28m30.867442116s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.061702074 (541h28m30.932785119s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091674256 (541h28m30.962757301s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091697750 (541h28m30.962780795s) [seg_pos={0 0}, upload=474844]: Segment reached optimal threshold of 13 pieces. Dec 28 04:57:31.091956844 (541h28m30.963039889s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.091967544 (541h28m30.963050589s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.091992962 (541h28m30.963075998s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092001779 (541h28m30.963084824s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092171182 (541h28m30.963254227s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092177233 (541h28m30.963260278s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092200828 (541h28m30.963283863s) [seg_pos={0 0}, upload=474844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:57:31.092277013 (541h28m30.963360058s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:57:31.092282724 (541h28m30.963365769s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:57:31.376618933 (541h28m31.247701978s) [upload=474844]: Done uploading segments... Dec 28 04:57:31.376642698 (541h28m31.247725743s) [upload=474844]: Upload finished. err: <nil> Duration: 6.128162854s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [..XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XXX] duration: 5.003085518s trace: https://jaeger.dev.storj.io/trace/4440c15a9a6419aa FAIL (2025-12-28 05:00:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.003116165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X................................................X.....] duration: 24.823551505s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 05:00:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 24.823593746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.012795299s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 05:00:00 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.012820215s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......XX] duration: 5.001012039s trace: https://jaeger.dev.storj.io/trace/32d28aab56816508 FAIL (2025-12-28 04:59:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001036055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX..] duration: 5.991519453s trace: https://jaeger.dev.storj.io/trace/4cd911d6d56e2fc8 FAIL (2025-12-28 04:59:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A) context deadline exceeded ======LOGS====== Dec 28 04:57:28.451800538 (541h28m28.322883604s) [upload=474844]: Starting upload Dec 28 04:57:28.451928753 (541h28m28.323011818s) [upload=474844]: Done starting upload Dec 28 04:57:28.451995079 (541h28m28.323078114s) [upload=474844]: Uploading segments... Dec 28 04:57:28.452185842 (541h28m28.323268887s) [upload=474844]: Got next segment. Inline: false Dec 28 04:57:28.452234825 (541h28m28.32331787s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:57:28.452240607 (541h28m28.323323642s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:57:28.695520939 (541h28m28.566603974s) [seg_pos={0 0}, upload=474844]: Begin upload segment... Dec 28 04:57:28.695571014 (541h28m28.566654049s) [seg_pos={0 0}, upload=474844]: Done begin upload segment. Dec 28 04:57:28.695588708 (541h28m28.566671753s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.695600470 (541h28m28.566683505s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696371448 (541h28m28.567454493s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696796467 (541h28m28.567879502s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.697682154 (541h28m28.568765199s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.698021711 (541h28m28.569104746s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699043126 (541h28m28.570126161s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699080727 (541h28m28.570163772s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700036667 (541h28m28.571119702s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700097854 (541h28m28.571180889s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.701122475 (541h28m28.57220552s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.702295218 (541h28m28.573378253s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.703764426 (541h28m28.574847471s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.704742959 (541h28m28.575826004s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.705869163 (541h28m28.576952208s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.706067191 (541h28m28.577150226s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:30.344383692 (541h28m30.215466727s) [upload=474844]: Next returned nil segment Dec 28 04:57:30.344408439 (541h28m30.215491494s) [upload=474844]: Waiting for error group managing segments... Dec 28 04:57:30.705618958 (541h28m30.576701993s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.710373342 (541h28m30.581456387s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.717849567 (541h28m30.588932612s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731461376 (541h28m30.602544421s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731614708 (541h28m30.602697753s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.737546954 (541h28m30.608629989s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748258019 (541h28m30.619341064s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748985264 (541h28m30.620068309s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.775426382 (541h28m30.646509427s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.957930196 (541h28m30.829013241s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.996359071 (541h28m30.867442116s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.061702074 (541h28m30.932785119s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091674256 (541h28m30.962757301s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091697750 (541h28m30.962780795s) [seg_pos={0 0}, upload=474844]: Segment reached optimal threshold of 13 pieces. Dec 28 04:57:31.091956844 (541h28m30.963039889s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.091967544 (541h28m30.963050589s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.091992962 (541h28m30.963075998s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092001779 (541h28m30.963084824s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092171182 (541h28m30.963254227s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092177233 (541h28m30.963260278s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092200828 (541h28m30.963283863s) [seg_pos={0 0}, upload=474844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:57:31.092277013 (541h28m30.963360058s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:57:31.092282724 (541h28m30.963365769s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:57:31.376618933 (541h28m31.247701978s) [upload=474844]: Done uploading segments... Dec 28 04:57:31.376642698 (541h28m31.247725743s) [upload=474844]: Upload finished. err: <nil> Duration: 5.991548981s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.213452707s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:59:15 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.213476833s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X................................................X....] duration: 41.514854943s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 04:59:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 41.514909166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.047670035s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:58:49 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.047697869s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......X] duration: 5.001233034s trace: https://jaeger.dev.storj.io/trace/23cadd85135f5f71 FAIL (2025-12-28 04:58:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001254263s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [XX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX.] duration: 28.984778135s trace: https://jaeger.dev.storj.io/trace/4cd911d6d56e2fc8 FAIL (2025-12-28 04:58:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A) context deadline exceeded ======LOGS====== Dec 28 04:57:28.451800538 (541h28m28.322883604s) [upload=474844]: Starting upload Dec 28 04:57:28.451928753 (541h28m28.323011818s) [upload=474844]: Done starting upload Dec 28 04:57:28.451995079 (541h28m28.323078114s) [upload=474844]: Uploading segments... Dec 28 04:57:28.452185842 (541h28m28.323268887s) [upload=474844]: Got next segment. Inline: false Dec 28 04:57:28.452234825 (541h28m28.32331787s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:57:28.452240607 (541h28m28.323323642s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:57:28.695520939 (541h28m28.566603974s) [seg_pos={0 0}, upload=474844]: Begin upload segment... Dec 28 04:57:28.695571014 (541h28m28.566654049s) [seg_pos={0 0}, upload=474844]: Done begin upload segment. Dec 28 04:57:28.695588708 (541h28m28.566671753s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.695600470 (541h28m28.566683505s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696371448 (541h28m28.567454493s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696796467 (541h28m28.567879502s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.697682154 (541h28m28.568765199s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.698021711 (541h28m28.569104746s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699043126 (541h28m28.570126161s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699080727 (541h28m28.570163772s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700036667 (541h28m28.571119702s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700097854 (541h28m28.571180889s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.701122475 (541h28m28.57220552s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.702295218 (541h28m28.573378253s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.703764426 (541h28m28.574847471s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.704742959 (541h28m28.575826004s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.705869163 (541h28m28.576952208s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.706067191 (541h28m28.577150226s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:30.344383692 (541h28m30.215466727s) [upload=474844]: Next returned nil segment Dec 28 04:57:30.344408439 (541h28m30.215491494s) [upload=474844]: Waiting for error group managing segments... Dec 28 04:57:30.705618958 (541h28m30.576701993s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.710373342 (541h28m30.581456387s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.717849567 (541h28m30.588932612s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731461376 (541h28m30.602544421s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731614708 (541h28m30.602697753s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.737546954 (541h28m30.608629989s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748258019 (541h28m30.619341064s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748985264 (541h28m30.620068309s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.775426382 (541h28m30.646509427s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.957930196 (541h28m30.829013241s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.996359071 (541h28m30.867442116s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.061702074 (541h28m30.932785119s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091674256 (541h28m30.962757301s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091697750 (541h28m30.962780795s) [seg_pos={0 0}, upload=474844]: Segment reached optimal threshold of 13 pieces. Dec 28 04:57:31.091956844 (541h28m30.963039889s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.091967544 (541h28m30.963050589s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.091992962 (541h28m30.963075998s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092001779 (541h28m30.963084824s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092171182 (541h28m30.963254227s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092177233 (541h28m30.963260278s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092200828 (541h28m30.963283863s) [seg_pos={0 0}, upload=474844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:57:31.092277013 (541h28m30.963360058s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:57:31.092282724 (541h28m30.963365769s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:57:31.376618933 (541h28m31.247701978s) [upload=474844]: Done uploading segments... Dec 28 04:57:31.376642698 (541h28m31.247725743s) [upload=474844]: Upload finished. err: <nil> Duration: 28.9848062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X......X................................................X...] duration: 46.326260904s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 04:58:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 46.326276082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.269728756s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:58:02 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.269759874s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXXX] duration: 30.004067258s trace: https://jaeger.dev.storj.io/trace/4cd911d6d56e2fc8 FAIL (2025-12-28 04:57:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A) context deadline exceeded ======LOGS====== Dec 28 04:57:28.451800538 (541h28m28.322883604s) [upload=474844]: Starting upload Dec 28 04:57:28.451928753 (541h28m28.323011818s) [upload=474844]: Done starting upload Dec 28 04:57:28.451995079 (541h28m28.323078114s) [upload=474844]: Uploading segments... Dec 28 04:57:28.452185842 (541h28m28.323268887s) [upload=474844]: Got next segment. Inline: false Dec 28 04:57:28.452234825 (541h28m28.32331787s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:57:28.452240607 (541h28m28.323323642s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:57:28.695520939 (541h28m28.566603974s) [seg_pos={0 0}, upload=474844]: Begin upload segment... Dec 28 04:57:28.695571014 (541h28m28.566654049s) [seg_pos={0 0}, upload=474844]: Done begin upload segment. Dec 28 04:57:28.695588708 (541h28m28.566671753s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.695600470 (541h28m28.566683505s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696371448 (541h28m28.567454493s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.696796467 (541h28m28.567879502s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.697682154 (541h28m28.568765199s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.698021711 (541h28m28.569104746s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699043126 (541h28m28.570126161s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.699080727 (541h28m28.570163772s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700036667 (541h28m28.571119702s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.700097854 (541h28m28.571180889s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.701122475 (541h28m28.57220552s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.702295218 (541h28m28.573378253s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.703764426 (541h28m28.574847471s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.704742959 (541h28m28.575826004s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.705869163 (541h28m28.576952208s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:28.706067191 (541h28m28.577150226s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Uploading piece... Dec 28 04:57:30.344383692 (541h28m30.215466727s) [upload=474844]: Next returned nil segment Dec 28 04:57:30.344408439 (541h28m30.215491494s) [upload=474844]: Waiting for error group managing segments... Dec 28 04:57:30.705618958 (541h28m30.576701993s) [noise=true , address=145.79.227.39:4014 , piece_id=MS2BHBIJLNV2XCDPIDBKIE32CXL6FITR75ZCN34Z4OK47GIJEQSA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.710373342 (541h28m30.581456387s) [noise=true , address=145.79.227.14:4840 , piece_id=LN24HVUIZSJDB3DZOGLKA253UPGIRZ34GPADJ7HZ3COKNVAW5KGA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.717849567 (541h28m30.588932612s) [noise=true , address=145.79.227.16:4910 , piece_id=JSFYX76JED242MHACQLVIB4PMIRLZCRWJBVHXCVKUXGJYPBCXPJQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731461376 (541h28m30.602544421s) [noise=true , address=145.79.227.37:4566 , piece_id=APVX2A6ARVWPHB5ZM5XXR6V3T5QKRRTQGWES3PAZK5ZRB42T55SA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.731614708 (541h28m30.602697753s) [noise=true , address=145.79.227.11:4729 , piece_id=FI7U2ZN3TKUOD2VZCZNKQCPZ6N3H42LFUM33ZVJMAZ3KPG3N3XQQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.737546954 (541h28m30.608629989s) [noise=true , address=145.79.227.45:4230 , piece_id=IWYKVP7YJRV6FJSKPNXVPEM6LDXIXAWIWJSDRLKEPJYJG43QOCSQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748258019 (541h28m30.619341064s) [noise=true , address=145.79.227.52:4446 , piece_id=RLPCRFH5B3H7DWE355AUOHSOP7MG5M6VEYO5N7MLMQV3XHGAAOMA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.748985264 (541h28m30.620068309s) [noise=true , address=145.79.227.47:4265 , piece_id=XCYDZ3LIYM5YXJ3I4MXHWNFBI3VPY3URZI65RUO2PWGJQFZRAS3Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.775426382 (541h28m30.646509427s) [noise=true , address=145.79.227.19:4079 , piece_id=FJMFDIYN745CZBJ6QMY5TF4BE5K3B4I4QNVRJ3ZS6AMSZ7T3EACA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.957930196 (541h28m30.829013241s) [noise=true , address=45.91.116.12:4225 , piece_id=HJUHLIGGEZ4QTC3LKFZJ3AAHCG2E7VBJE7S473QFWQXZSI3ERMCA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:30.996359071 (541h28m30.867442116s) [noise=true , address=77.90.31.13:4627 , piece_id=U5EMLJAEWFAAMA2VBS3EEKFM6AJCTCQBQ2QBQ45W4KWFE7SSR5HQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.061702074 (541h28m30.932785119s) [noise=true , address=77.90.31.11:4555 , piece_id=GSTGG7Z4Y3FRMJ37HKDTYYSSRVVQSXLOXECW32M75WOQUVF2737A, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091674256 (541h28m30.962757301s) [noise=true , address=45.91.116.8:4046 , piece_id=A2WA2C6XPLCPLWQDDYLMVO6JUTVHC5B2VKE6PQNUSIM3IVBPAI5Q, seg_pos={0 0}, upload=474844]: Done uploading piece. err: <nil> Dec 28 04:57:31.091697750 (541h28m30.962780795s) [seg_pos={0 0}, upload=474844]: Segment reached optimal threshold of 13 pieces. Dec 28 04:57:31.091956844 (541h28m30.963039889s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.091967544 (541h28m30.963050589s) [noise=true , address=77.90.31.21:4878 , piece_id=ZBIBD66UJR4SKBS6DLZQPQBUS6Z52B22UIDK64YQ6ZXNR3G4E7AQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.091992962 (541h28m30.963075998s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092001779 (541h28m30.963084824s) [noise=true , address=45.91.116.5:4300 , piece_id=RDJX6OAQKMGZOYZUZ6TKV3A4PHCXGVROJDMCBYOBCY4UQ7HHQNOQ, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092171182 (541h28m30.963254227s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 28 04:57:31.092177233 (541h28m30.963260278s) [noise=true , address=145.79.227.44:4195 , piece_id=E3ETRRVTSYQOQCP2YXOOFF55WPANJEBMM6HI3SH3WQ2OO2S7CXOA, seg_pos={0 0}, upload=474844]: Long tail cancellation due to optimal threshold Dec 28 04:57:31.092200828 (541h28m30.963283863s) [seg_pos={0 0}, upload=474844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:57:31.092277013 (541h28m30.963360058s) [upload=474844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:57:31.092282724 (541h28m30.963365769s) [upload=474844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:57:31.376618933 (541h28m31.247701978s) [upload=474844]: Done uploading segments... Dec 28 04:57:31.376642698 (541h28m31.247725743s) [upload=474844]: Upload finished. err: <nil> Duration: 30.004090541s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X......X................................................X..] duration: 52.120177225s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 04:57:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 52.120193847s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.......] duration: 3.517932364s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:57:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517953964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.147930184s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:57:34 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.147958267s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XXX] duration: 30.002161041s trace: https://jaeger.dev.storj.io/trace/534c43dc58fae9e0 FAIL (2025-12-28 04:57:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== gments... Dec 28 04:56:40.981021111 (541h27m40.852104156s) [upload=474834]: Got next segment. Inline: false Dec 28 04:56:40.981049044 (541h27m40.852132079s) [upload=474834]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:56:40.981053092 (541h27m40.852136127s) [upload=474834]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:56:41.227027684 (541h27m41.098110739s) [seg_pos={0 0}, upload=474834]: Begin upload segment... Dec 28 04:56:41.227084923 (541h27m41.098167968s) [seg_pos={0 0}, upload=474834]: Done begin upload segment. Dec 28 04:56:41.227122324 (541h27m41.098205359s) [noise=true , address=145.79.227.27:4691 , piece_id=WAJCFDNSLUBE54MGTNMCPSKATWEVWFNNYWI3R5LHMIXPX7NOSOYQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.227129869 (541h27m41.098212924s) [noise=true , address=45.91.116.11:4119 , piece_id=HU32YN4OSCWMUJTH2VTZQE2QXNL3QRMYAL6L52KEMIW5MV2G74FQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.227629009 (541h27m41.098712054s) [noise=true , address=145.79.227.37:4566 , piece_id=RABW3MU64YTVCQB5BZY6XUUFF5R6L27G7WE6AKRYZ5ZTFLUIXJ7A, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.228053758 (541h27m41.099136803s) [noise=true , address=45.91.116.12:4227 , piece_id=GSXYGAIHYXIMF7BWAUBZIBFTUX5DB5LQB2GJUIGW75XE33DM3WXQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.228480770 (541h27m41.099563816s) [noise=true , address=145.79.227.5:4222 , piece_id=RFLIP6Z5VMN4S3HMFOMAGEE5KBQNFIX6B4B3PQGBGFCABWIOPR6Q, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.229405762 (541h27m41.100488807s) [noise=true , address=45.91.116.10:4081 , piece_id=QA5A73OZBVQFS732MIBCBO6TE2NU4RY6UEZJRQU2C7OJ7MGZ2CAQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.229785485 (541h27m41.10086852s) [noise=true , address=45.91.116.9:4261 , piece_id=5ECT436ZVUAQQOVCRBGSYTHC6SNENOKP4N7C3PNVVGCMVZFMX2HQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.230182631 (541h27m41.101265676s) [noise=true , address=45.91.116.8:4045 , piece_id=M2WLLT7SAUK3UEG4Z3BB7NODPKLZL3QIMB4QLPTN5KTF4F4BCWCA, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.231349713 (541h27m41.102432749s) [noise=true , address=45.91.116.13:4154 , piece_id=2SF7DISCMAISR4VMDUZ24OSWZ3JA5F5NG5CQ3CEMY4R2G6GNOA3A, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.232537556 (541h27m41.103620591s) [noise=true , address=145.79.227.14:4840 , piece_id=DPCWONW6LKAK4OKMG5RN3UV3YAG3RVP6WSJJDJYLYEG43265K24Q, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.233671354 (541h27m41.104754399s) [noise=true , address=77.90.31.15:4697 , piece_id=U5XDAZJJFY57KX267WDN3HYSIH6DOGK5DD3JJLBWUFSQUWWTHVBA, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.235109243 (541h27m41.106192278s) [noise=true , address=145.79.227.17:4971 , piece_id=CYZDA6CI5XMHMMXVMCVKOW3AXPZXPWGLHQ2L5WX35PIWM3D4JRUQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.236492216 (541h27m41.107575251s) [noise=true , address=145.79.227.50:4374 , piece_id=42QSBJTZMMVLYOB75ACLP6KKLX3NTB6N5TADCNRBB2SMPYYQOVOA, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.237888194 (541h27m41.108971229s) [noise=true , address=77.90.31.14:4663 , piece_id=SSDCZJQB4CISX4NMLOQAEOS2OHCKKW2V2Q33Z3HWTXPFOEHYHLTQ, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.239095343 (541h27m41.110178388s) [noise=true , address=145.79.227.2:4114 , piece_id=SQCA3M6GXUVTNFA7OMBBLIXTLQVQ2FKMP2SA7Y2UHC5VRQWY7I4Q, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:41.239653235 (541h27m41.11073628s) [noise=true , address=77.90.31.21:4878 , piece_id=3RQ2QIKITNFDWEJYHMJMFACJSYQA57ZLMWDH374GSVMVCDPOYBSA, seg_pos={0 0}, upload=474834]: Uploading piece... Dec 28 04:56:42.933914392 (541h27m42.804997437s) [upload=474834]: Next returned nil segment Dec 28 04:56:42.933934900 (541h27m42.805017936s) [upload=474834]: Waiting for error group managing segments... Dec 28 04:56:43.224234625 (541h27m43.09531767s) [noise=true , address=145.79.227.27:4691 , piece_id=WAJCFDNSLUBE54MGTNMCPSKATWEVWFNNYWI3R5LHMIXPX7NOSOYQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.229268190 (541h27m43.100351225s) [noise=true , address=145.79.227.5:4222 , piece_id=RFLIP6Z5VMN4S3HMFOMAGEE5KBQNFIX6B4B3PQGBGFCABWIOPR6Q, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.230062331 (541h27m43.101145377s) [noise=true , address=145.79.227.37:4566 , piece_id=RABW3MU64YTVCQB5BZY6XUUFF5R6L27G7WE6AKRYZ5ZTFLUIXJ7A, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.241520139 (541h27m43.112603184s) [noise=true , address=145.79.227.2:4114 , piece_id=SQCA3M6GXUVTNFA7OMBBLIXTLQVQ2FKMP2SA7Y2UHC5VRQWY7I4Q, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.245773718 (541h27m43.116856753s) [noise=true , address=145.79.227.14:4840 , piece_id=DPCWONW6LKAK4OKMG5RN3UV3YAG3RVP6WSJJDJYLYEG43265K24Q, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.249772373 (541h27m43.120855408s) [noise=true , address=145.79.227.17:4971 , piece_id=CYZDA6CI5XMHMMXVMCVKOW3AXPZXPWGLHQ2L5WX35PIWM3D4JRUQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.263589753 (541h27m43.134672798s) [noise=true , address=145.79.227.50:4374 , piece_id=42QSBJTZMMVLYOB75ACLP6KKLX3NTB6N5TADCNRBB2SMPYYQOVOA, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.410124066 (541h27m43.281207102s) [noise=true , address=77.90.31.21:4878 , piece_id=3RQ2QIKITNFDWEJYHMJMFACJSYQA57ZLMWDH374GSVMVCDPOYBSA, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.431508726 (541h27m43.302591761s) [noise=true , address=77.90.31.15:4697 , piece_id=U5XDAZJJFY57KX267WDN3HYSIH6DOGK5DD3JJLBWUFSQUWWTHVBA, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.523009627 (541h27m43.394092672s) [noise=true , address=77.90.31.14:4663 , piece_id=SSDCZJQB4CISX4NMLOQAEOS2OHCKKW2V2Q33Z3HWTXPFOEHYHLTQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.588758443 (541h27m43.459841488s) [noise=true , address=45.91.116.10:4081 , piece_id=QA5A73OZBVQFS732MIBCBO6TE2NU4RY6UEZJRQU2C7OJ7MGZ2CAQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.602212371 (541h27m43.473295406s) [noise=true , address=45.91.116.9:4261 , piece_id=5ECT436ZVUAQQOVCRBGSYTHC6SNENOKP4N7C3PNVVGCMVZFMX2HQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.633373786 (541h27m43.504456821s) [noise=true , address=45.91.116.12:4227 , piece_id=GSXYGAIHYXIMF7BWAUBZIBFTUX5DB5LQB2GJUIGW75XE33DM3WXQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: <nil> Dec 28 04:56:43.633391681 (541h27m43.504474716s) [seg_pos={0 0}, upload=474834]: Segment reached optimal threshold of 13 pieces. Dec 28 04:56:43.633588164 (541h27m43.504671199s) [noise=true , address=45.91.116.11:4119 , piece_id=HU32YN4OSCWMUJTH2VTZQE2QXNL3QRMYAL6L52KEMIW5MV2G74FQ, seg_pos={0 0}, upload=474834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:39374->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:39374->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39374->45.91.116.11:4119: use of closed network connection Dec 28 04:56:43.633595048 (541h27m43.504678083s) [noise=true , address=45.91.116.11:4119 , piece_id=HU32YN4OSCWMUJTH2VTZQE2QXNL3QRMYAL6L52KEMIW5MV2G74FQ, seg_pos={0 0}, upload=474834]: Long tail cancellation due to optimal threshold Dec 28 04:56:43.633859120 (541h27m43.504942156s) [noise=true , address=45.91.116.13:4154 , piece_id=2SF7DISCMAISR4VMDUZ24OSWZ3JA5F5NG5CQ3CEMY4R2G6GNOA3A, seg_pos={0 0}, upload=474834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 04:56:43.633865643 (541h27m43.504948678s) [noise=true , address=45.91.116.13:4154 , piece_id=2SF7DISCMAISR4VMDUZ24OSWZ3JA5F5NG5CQ3CEMY4R2G6GNOA3A, seg_pos={0 0}, upload=474834]: Long tail cancellation due to optimal threshold Dec 28 04:56:43.633879099 (541h27m43.504962134s) [noise=true , address=45.91.116.8:4045 , piece_id=M2WLLT7SAUK3UEG4Z3BB7NODPKLZL3QIMB4QLPTN5KTF4F4BCWCA, seg_pos={0 0}, upload=474834]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:39190->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:39190->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39190->45.91.116.8:4045: use of closed network connection Dec 28 04:56:43.633894919 (541h27m43.504977954s) [noise=true , address=45.91.116.8:4045 , piece_id=M2WLLT7SAUK3UEG4Z3BB7NODPKLZL3QIMB4QLPTN5KTF4F4BCWCA, seg_pos={0 0}, upload=474834]: Long tail cancellation due to optimal threshold Dec 28 04:56:43.633917813 (541h27m43.505000858s) [seg_pos={0 0}, upload=474834]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:56:43.634018354 (541h27m43.505101389s) [upload=474834]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:56:43.634026109 (541h27m43.505109164s) [upload=474834]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:56:43.931901748 (541h27m43.802984783s) [upload=474834]: Done uploading segments... Dec 28 04:56:43.931919301 (541h27m43.803002346s) [upload=474834]: Upload finished. err: <nil> Duration: 30.002182152s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX......] duration: 4.937736562s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:57:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.937767131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX.....] duration: 5.001493743s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:56:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001513s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X......X................................................X.] duration: 50.027071675s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 04:56:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 50.02709491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.526283975s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:56:30 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.526309817s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.XX] duration: 30.002981257s trace: https://jaeger.dev.storj.io/trace/6252e020c7ae4d89 FAIL (2025-12-28 04:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 7NZAVKMTTO6BHQHGDDDLQWKD2LVPRKQFOANHITDQ55HTBJ3DCPNA) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: C5MX4JEHM5OBAXLZ4UKHG7UOXIGYK7QC6VYI6X326JC2J52JERUQ) context deadline exceeded ======LOGS====== alse Dec 28 04:55:53.298917644 (541h26m53.170000679s) [upload=474824]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:55:53.298922754 (541h26m53.170005789s) [upload=474824]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:55:53.546364239 (541h26m53.417447294s) [seg_pos={0 0}, upload=474824]: Begin upload segment... Dec 28 04:55:53.546435535 (541h26m53.41751858s) [seg_pos={0 0}, upload=474824]: Done begin upload segment. Dec 28 04:55:53.546496060 (541h26m53.417579105s) [noise=true , address=145.79.227.45:4229 , piece_id=5LJHLM7N2IY5BKSIL73HKDXJE7WZICPTAPIFH3KLITG7EQVFHAJQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.546526308 (541h26m53.417609353s) [noise=true , address=145.79.227.17:4970 , piece_id=HOAB5SDSPXOJW3UIQBXV3UFOYB4YTYBIIFVQT2RY4YSXHVZ4BHNA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.547393028 (541h26m53.418476084s) [noise=true , address=145.79.227.18:4007 , piece_id=RH3FL4JEF6XCSRNI2CISU6ZM5UP6H7TBURNYW4UA2E3JUIIVUM2A, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.547558232 (541h26m53.418641267s) [noise=true , address=145.79.227.48:4303 , piece_id=7A4W64TA2SL7Y3MW5DLONROFDTV7VKH27JDLX2LJSZZV6GPAZ7AQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.547963595 (541h26m53.41904663s) [noise=true , address=45.91.116.11:4118 , piece_id=HK76BTBHT3VNXG4UH2JJS3GUAXUYVURU2BCPFN6QLDCCQWKLH6LQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.548336084 (541h26m53.419419119s) [noise=true , address=45.91.116.7:4012 , piece_id=Y7DOVLHMHJN435DEVNNDYUM5HUC7WVF4SZVK5SQ7LZ2P4LTYYK4Q, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.548728531 (541h26m53.419811566s) [noise=true , address=45.91.116.12:4226 , piece_id=4DNQK7JB4LFFUQNO5ZPE3KKFJRXD66YS5GRTFVDZMYV7IAVSHVMQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.550483803 (541h26m53.421566838s) [noise=true , address=45.91.116.9:4261 , piece_id=NVQX6UYVAXLM24RFJSGZAE532UI2WXSWFK3ZWAFBGUYRVSUC6H3Q, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.551201560 (541h26m53.422284595s) [noise=true , address=145.79.227.46:4602 , piece_id=NCWARPAC6X7WTLGNY6GBW7MCVZVY3LNYYMPIVB52LSY4BSHQOTRA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.552381968 (541h26m53.423465003s) [noise=true , address=145.79.227.47:4267 , piece_id=G5DQYRMN2ATKS7FFARBYYQ7K45K2FIKDZ5AWZWKFQU2ALGKVWUVA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.553527800 (541h26m53.424610845s) [noise=true , address=45.91.116.13:4154 , piece_id=7NZAVKMTTO6BHQHGDDDLQWKD2LVPRKQFOANHITDQ55HTBJ3DCPNA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.554933717 (541h26m53.426016752s) [noise=true , address=77.90.31.14:4662 , piece_id=C5MX4JEHM5OBAXLZ4UKHG7UOXIGYK7QC6VYI6X326JC2J52JERUQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.556069299 (541h26m53.427152334s) [noise=true , address=45.91.116.10:4083 , piece_id=BABNZEZXRQIXYAPEZEQGHHHWSQWHVEE4PBP6BKG3BW6RFNOYJBRA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.557465778 (541h26m53.428548813s) [noise=true , address=145.79.227.51:4409 , piece_id=FO4PQ7U3NZ7UHUFXN3XXKJBRRLJKNEXTLHRBJWDH2263EUE23MVA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.559189119 (541h26m53.430272164s) [noise=true , address=145.79.227.14:4840 , piece_id=GYLS42B3O34UKR2TUI2XXDA7HKPXZCCFSFLSKIM3MLXV6MKWDLSQ, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:53.560334921 (541h26m53.431417966s) [noise=true , address=145.79.227.24:4583 , piece_id=3HEUI5DJQG3LYSPOHJWMBHCAVEH7GXQPQURP53FTSBWOCR47C3SA, seg_pos={0 0}, upload=474824]: Uploading piece... Dec 28 04:55:55.233212892 (541h26m55.104295928s) [upload=474824]: Next returned nil segment Dec 28 04:55:55.233232870 (541h26m55.104315915s) [upload=474824]: Waiting for error group managing segments... Dec 28 04:55:55.556686274 (541h26m55.427769319s) [noise=true , address=145.79.227.17:4970 , piece_id=HOAB5SDSPXOJW3UIQBXV3UFOYB4YTYBIIFVQT2RY4YSXHVZ4BHNA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.558080840 (541h26m55.429163885s) [noise=true , address=145.79.227.46:4602 , piece_id=NCWARPAC6X7WTLGNY6GBW7MCVZVY3LNYYMPIVB52LSY4BSHQOTRA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.559563834 (541h26m55.430646869s) [noise=true , address=145.79.227.45:4229 , piece_id=5LJHLM7N2IY5BKSIL73HKDXJE7WZICPTAPIFH3KLITG7EQVFHAJQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.563317130 (541h26m55.434400175s) [noise=true , address=145.79.227.18:4007 , piece_id=RH3FL4JEF6XCSRNI2CISU6ZM5UP6H7TBURNYW4UA2E3JUIIVUM2A, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.567219010 (541h26m55.438302045s) [noise=true , address=145.79.227.48:4303 , piece_id=7A4W64TA2SL7Y3MW5DLONROFDTV7VKH27JDLX2LJSZZV6GPAZ7AQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.575212431 (541h26m55.446295466s) [noise=true , address=145.79.227.47:4267 , piece_id=G5DQYRMN2ATKS7FFARBYYQ7K45K2FIKDZ5AWZWKFQU2ALGKVWUVA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.577744402 (541h26m55.448827437s) [noise=true , address=145.79.227.14:4840 , piece_id=GYLS42B3O34UKR2TUI2XXDA7HKPXZCCFSFLSKIM3MLXV6MKWDLSQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.583893000 (541h26m55.454976045s) [noise=true , address=145.79.227.24:4583 , piece_id=3HEUI5DJQG3LYSPOHJWMBHCAVEH7GXQPQURP53FTSBWOCR47C3SA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.593375135 (541h26m55.46445818s) [noise=true , address=145.79.227.51:4409 , piece_id=FO4PQ7U3NZ7UHUFXN3XXKJBRRLJKNEXTLHRBJWDH2263EUE23MVA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.835459841 (541h26m55.706542876s) [noise=true , address=77.90.31.14:4662 , piece_id=C5MX4JEHM5OBAXLZ4UKHG7UOXIGYK7QC6VYI6X326JC2J52JERUQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.854465240 (541h26m55.725548275s) [noise=true , address=45.91.116.12:4226 , piece_id=4DNQK7JB4LFFUQNO5ZPE3KKFJRXD66YS5GRTFVDZMYV7IAVSHVMQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.890042224 (541h26m55.761125269s) [noise=true , address=45.91.116.9:4261 , piece_id=NVQX6UYVAXLM24RFJSGZAE532UI2WXSWFK3ZWAFBGUYRVSUC6H3Q, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.923662743 (541h26m55.794745788s) [noise=true , address=45.91.116.13:4154 , piece_id=7NZAVKMTTO6BHQHGDDDLQWKD2LVPRKQFOANHITDQ55HTBJ3DCPNA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: <nil> Dec 28 04:55:55.923687490 (541h26m55.794770545s) [seg_pos={0 0}, upload=474824]: Segment reached optimal threshold of 13 pieces. Dec 28 04:55:55.923958245 (541h26m55.79504129s) [noise=true , address=45.91.116.11:4118 , piece_id=HK76BTBHT3VNXG4UH2JJS3GUAXUYVURU2BCPFN6QLDCCQWKLH6LQ, seg_pos={0 0}, upload=474824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: read tcp 188.166.225.170:57286->45.91.116.11:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:55:55.923968545 (541h26m55.79505159s) [noise=true , address=45.91.116.11:4118 , piece_id=HK76BTBHT3VNXG4UH2JJS3GUAXUYVURU2BCPFN6QLDCCQWKLH6LQ, seg_pos={0 0}, upload=474824]: Long tail cancellation due to optimal threshold Dec 28 04:55:55.923979085 (541h26m55.79506212s) [noise=true , address=45.91.116.7:4012 , piece_id=Y7DOVLHMHJN435DEVNNDYUM5HUC7WVF4SZVK5SQ7LZ2P4LTYYK4Q, seg_pos={0 0}, upload=474824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:60390->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:60390->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60390->45.91.116.7:4012: use of closed network connection Dec 28 04:55:55.923986890 (541h26m55.795069935s) [noise=true , address=45.91.116.7:4012 , piece_id=Y7DOVLHMHJN435DEVNNDYUM5HUC7WVF4SZVK5SQ7LZ2P4LTYYK4Q, seg_pos={0 0}, upload=474824]: Long tail cancellation due to optimal threshold Dec 28 04:55:55.924119062 (541h26m55.795202107s) [noise=true , address=45.91.116.10:4083 , piece_id=BABNZEZXRQIXYAPEZEQGHHHWSQWHVEE4PBP6BKG3BW6RFNOYJBRA, seg_pos={0 0}, upload=474824]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:49262->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:49262->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49262->45.91.116.10:4083: use of closed network connection Dec 28 04:55:55.924127517 (541h26m55.795210553s) [noise=true , address=45.91.116.10:4083 , piece_id=BABNZEZXRQIXYAPEZEQGHHHWSQWHVEE4PBP6BKG3BW6RFNOYJBRA, seg_pos={0 0}, upload=474824]: Long tail cancellation due to optimal threshold Dec 28 04:55:55.924231075 (541h26m55.79531411s) [seg_pos={0 0}, upload=474824]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:55:55.924317610 (541h26m55.795400655s) [upload=474824]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:55:55.924324413 (541h26m55.795407448s) [upload=474824]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:55:56.212721053 (541h26m56.083804098s) [upload=474824]: Done uploading segments... Dec 28 04:55:56.212740581 (541h26m56.083823616s) [upload=474824]: Upload finished. err: <nil> Duration: 30.00300423s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [..X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX....] duration: 3.491236959s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:56:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.491266995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 7.876117013s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:55:51 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 7.876141458s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X......X................................................X] duration: 1m0.004690157s trace: https://jaeger.dev.storj.io/trace/ad05144256a327 FAIL (2025-12-28 04:55:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.205114168 (541h25m44.076197213s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.206656344 (541h25m44.07773938s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207072517 (541h25m44.078155562s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207439956 (541h25m44.078523001s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.207808278 (541h25m44.078891323s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208218329 (541h25m44.079301364s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.208789887 (541h25m44.079872932s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210114719 (541h25m44.081197765s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.210794063 (541h25m44.081877098s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.211639383 (541h25m44.082722418s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212103817 (541h25m44.083186862s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.212601635 (541h25m44.08368467s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213048576 (541h25m44.084131621s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213505846 (541h25m44.084588891s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:44.213973908 (541h25m44.085056953s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Uploading piece... Dec 28 04:54:45.647080259 (541h25m45.518163294s) [upload=474810]: Next returned nil segment Dec 28 04:54:45.647093183 (541h25m45.518176218s) [upload=474810]: Waiting for error group managing segments... Dec 28 04:54:46.062185451 (541h25m45.933268506s) [noise=true , address=145.79.227.46:4622 , piece_id=R2UZUWJH6TQAQFQNR4MZMUNMSBAGVXBBWCDQBALXIXR2CFLBCQJA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.079608828 (541h25m45.950691863s) [noise=true , address=145.79.227.8:4361 , piece_id=UVS3BFWAAFFHJOL3FTEECNXQRSXM377GAJFJ6IEFEXE72BY3TBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.100828373 (541h25m45.971911408s) [noise=true , address=145.79.227.34:4940 , piece_id=O5XZQYT5CCQCSEWDTQ5PIIFOSM7IWVIOOQFWQUZW2TSZFOQYYX3A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.143892373 (541h25m46.014975418s) [noise=true , address=145.79.227.29:4946 , piece_id=FELZGIB2ZCVOJ5HIDIJCIIIVY4QUWRQC65P73SXOZKISDRVYJ42A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.248149775 (541h25m46.11923281s) [noise=true , address=77.90.31.16:4745 , piece_id=KMW567REWG2EDNDDSTNRZYASPSOHEMUM7UT6UZ6SX4EAHAGVCLVQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.257170913 (541h25m46.128253958s) [noise=true , address=77.90.31.13:4636 , piece_id=XT47MABC6XAZVLFJAL6COCQ6AX4SFCG7DHCNX4GOHUZP2X7EP6OA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.318102596 (541h25m46.189185631s) [noise=true , address=145.79.227.40:4076 , piece_id=D5S2CKUZQMRSCLYOJSDTLWOFXFSDBOUGOD7OEBDOGPASWSZXUXLQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.357722798 (541h25m46.228805843s) [noise=true , address=77.90.31.11:4579 , piece_id=J7FZY2GGX2X2VULN3DVH6AX53HILGJTO24PNFYYX6FNHGDWGO6UA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.366858364 (541h25m46.237941399s) [noise=true , address=77.90.31.21:4911 , piece_id=RM3BHHGOEA3CMLRZI4DQZAGQVFU3EJWPEA7F5H7VRBUBIC5JVBIQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.371073199 (541h25m46.242156234s) [noise=true , address=77.90.31.17:4787 , piece_id=U72BIKPP2O4AGTW3LA5HFWX67X4RC7VQA5ULJBXWYGDXCVIIFSRQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.479419587 (541h25m46.350502622s) [noise=true , address=77.90.31.18:4838 , piece_id=K75IKROAOJQGCKVSVAJF42BPXRVDBL3VQTIVRKHU3GZ6XXXWTALA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:46.936181621 (541h25m46.807264667s) [noise=true , address=77.90.31.19:4874 , piece_id=NZNB6ARF2LZHJJIM6CBZUR2LDA6DTO6M57G35TYX6RWO225GTSHQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:47.726099554 (541h25m47.597182599s) [noise=true , address=77.90.31.15:4724 , piece_id=ONZQ4LZKLYDQBY4OD7ILGFKQYVQY3EF22WGEMU3YX6QPSUY2EHNA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168496383 (541h25m48.039579428s) [noise=true , address=145.79.227.30:4776 , piece_id=DOKH7DXQDJOU5T25EUGIWESYWJQRD5BZV5CR3NC3DI7FGJKAW77A, seg_pos={0 0}, upload=474810]: Done uploading piece. err: <nil> Dec 28 04:54:48.168527993 (541h25m48.039611028s) [seg_pos={0 0}, upload=474810]: Segment reached optimal threshold of 14 pieces. Dec 28 04:54:48.168767760 (541h25m48.039850805s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35450->64.49.13.169:4707: use of closed network connection Dec 28 04:54:48.168776306 (541h25m48.039859341s) [noise=true , address=64.49.13.169:4707 , piece_id=HB2TT6HNLYGULUPCIWIUNM4RXRQLGRIU7BX43F6ITCEAWSJMPJCA, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169027394 (541h25m48.040110439s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169035539 (541h25m48.040118574s) [noise=true , address=145.79.227.13:4809 , piece_id=WUGCPQCXZWDQVGCGECT7GFAAVY7EWBW44X4BIGKZUHZXO2LZF2JQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169178221 (541h25m48.040261266s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LV4vwjXQvjzxs1jtSpNXZ6VvzLXbQBCYQBn5gzb7jh2DD9qpi): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169186097 (541h25m48.040269132s) [noise=true , address=145.79.227.44:4214 , piece_id=GC3OMVXOKDYT4FNJXV4BYZDWE6SAZSQNYIG6ZZNXJVCL2BMDT5TQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169324280 (541h25m48.040407335s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF Dec 28 04:54:48.169332125 (541h25m48.04041516s) [noise=true , address=145.79.227.33:4881 , piece_id=V7FW2I3XQPUCUB4DVXSCS625GBSXHHCZE3UDWH5BOTE3IQSBQXGQ, seg_pos={0 0}, upload=474810]: Long tail cancellation due to optimal threshold Dec 28 04:54:48.169353565 (541h25m48.04043661s) [seg_pos={0 0}, upload=474810]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:54:48.169490636 (541h25m48.040573681s) [upload=474810]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:54:48.169498311 (541h25m48.040581346s) [upload=474810]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:54:48.459097541 (541h25m48.330180576s) [upload=474810]: Done uploading segments... Dec 28 04:54:48.459115965 (541h25m48.330199001s) [upload=474810]: Upload finished. err: <nil> Duration: 1m0.004703944s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [...X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX...] duration: 4.012328065s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:55:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.012379744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.X] duration: 30.250510569s trace: https://jaeger.dev.storj.io/trace/86cd76882404dbf FAIL (2025-12-28 04:55:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 28 04:54:49.653453884 (541h25m49.524536929s) [upload=474812]: Starting upload Dec 28 04:54:49.653597267 (541h25m49.524680312s) [upload=474812]: Done starting upload Dec 28 04:54:49.653658754 (541h25m49.524741799s) [upload=474812]: Uploading segments... Dec 28 04:54:49.654014451 (541h25m49.525097496s) [upload=474812]: Got next segment. Inline: false Dec 28 04:54:49.654076109 (541h25m49.525159154s) [upload=474812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:54:49.654082230 (541h25m49.525165275s) [upload=474812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:54:49.897536083 (541h25m49.768619128s) [seg_pos={0 0}, upload=474812]: Begin upload segment... Dec 28 04:54:49.897600455 (541h25m49.76868351s) [seg_pos={0 0}, upload=474812]: Done begin upload segment. Dec 28 04:54:49.897690196 (541h25m49.768773241s) [noise=true , address=77.90.31.15:4697 , piece_id=MXGUEVP3I7A6POAKNFCE6FGGVJ22W3X6HJAZ5FMEJREEJ6UMER5A, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.897699544 (541h25m49.768782589s) [noise=true , address=145.79.227.38:4977 , piece_id=VGWXTQUT3M5JJDTKJRQBXEDDQCBH5LL6SAVBBQOBADDXEKJHVKLA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.898899679 (541h25m49.769982724s) [noise=true , address=145.79.227.10:4403 , piece_id=J4BAJTPKS4Q35DNDKQ3AKL3PCAQMNP3CV6DWVGYNK3TKISR3O52A, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.899491887 (541h25m49.770574922s) [noise=true , address=145.79.227.48:4303 , piece_id=VRW6US5GNPTQAJP5CWPXRKUP234ULZ4SJHSJHILZYFZTKIM3KU6A, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.900689968 (541h25m49.771773013s) [noise=true , address=145.79.227.12:4765 , piece_id=QLSSBYEI76XZQHJBHMNFVRRRU4G234NZGSE23DZ3OORXEOEKGHGQ, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.901465465 (541h25m49.77254851s) [noise=true , address=145.79.227.37:4566 , piece_id=WULR2SAELC7UFQWDRHTQ7RD4QV2TWF2IWV6JR3LRJUCOJPDJLMPA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.903223973 (541h25m49.774307008s) [noise=true , address=145.79.227.51:4410 , piece_id=JHAO2QMP6UU4GBSYEGZOB67AXMMYL33ISL57RQLK4I7CQ54FKTWA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.903839766 (541h25m49.774922811s) [noise=true , address=145.79.227.46:4601 , piece_id=FFE47R5Z57XU56I2CE7DOO4DGJLEZ3T2GMSQA65UT67BU6IPBH3Q, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.903907024 (541h25m49.774990069s) [noise=true , address=145.79.227.29:4941 , piece_id=OKZVRGHADKX2TM6T7LX6WJM7NLKMRGYZLHAWAHU2HQMMMH2UOTMA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.905606039 (541h25m49.776689084s) [noise=true , address=145.79.227.45:4231 , piece_id=NG7VBSJ56VPOKU7ND54LHLKY6NZSSX6KXQUKPM2FUI4Q7MH3S34Q, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.906923738 (541h25m49.778006783s) [noise=true , address=145.79.227.5:4221 , piece_id=X4CCIJM4WBTPRD5VV5DY3R6HLVH3KVEZBHOR5UKCZU2J5F7ZA24Q, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.909545309 (541h25m49.780628354s) [noise=true , address=77.90.31.14:4664 , piece_id=ZHQWEUCQEKKKJBC7TOS6SNDXFHMJPZ3WR76O6ZRI4JBYDFAFYOZQ, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.911802538 (541h25m49.782885583s) [noise=true , address=77.90.31.19:4844 , piece_id=MHW7DAB6Q6QNXHHTCYN2LKGNX7OTLIIIOTC5KJB6VC4AAER7GULQ, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.914125171 (541h25m49.785208206s) [noise=true , address=145.79.227.49:4337 , piece_id=LXZMFTAQHBRHVEK3C4ZBHXI4PEZDY3I4AFZJY223MUJ2HRH7A2OA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.918405471 (541h25m49.789488516s) [noise=true , address=145.79.227.52:4447 , piece_id=EVLCICHGDCSN5GNKMNVNGEVD4PY6QOU2B6D7I5EH6EC5AOG2XWZA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:49.919805717 (541h25m49.790888752s) [noise=true , address=45.91.116.4:4939 , piece_id=IJTT33HWOA3VUKPQNRWXPBQNF6OPECF75RHJ5JI7PKFJXQAP7PFA, seg_pos={0 0}, upload=474812]: Uploading piece... Dec 28 04:54:51.787343730 (541h25m51.658426775s) [upload=474812]: Next returned nil segment Dec 28 04:54:51.787359349 (541h25m51.658442394s) [upload=474812]: Waiting for error group managing segments... Dec 28 04:54:52.087775788 (541h25m51.958858833s) [noise=true , address=77.90.31.15:4697 , piece_id=MXGUEVP3I7A6POAKNFCE6FGGVJ22W3X6HJAZ5FMEJREEJ6UMER5A, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:52.429143603 (541h25m52.300226638s) [noise=true , address=77.90.31.19:4844 , piece_id=MHW7DAB6Q6QNXHHTCYN2LKGNX7OTLIIIOTC5KJB6VC4AAER7GULQ, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:52.926529520 (541h25m52.797612555s) [noise=true , address=145.79.227.5:4221 , piece_id=X4CCIJM4WBTPRD5VV5DY3R6HLVH3KVEZBHOR5UKCZU2J5F7ZA24Q, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:53.048402815 (541h25m52.91948586s) [noise=true , address=45.91.116.4:4939 , piece_id=IJTT33HWOA3VUKPQNRWXPBQNF6OPECF75RHJ5JI7PKFJXQAP7PFA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:53.201991179 (541h25m53.073074214s) [noise=true , address=77.90.31.14:4664 , piece_id=ZHQWEUCQEKKKJBC7TOS6SNDXFHMJPZ3WR76O6ZRI4JBYDFAFYOZQ, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:54.036061573 (541h25m53.907144608s) [noise=true , address=145.79.227.49:4337 , piece_id=LXZMFTAQHBRHVEK3C4ZBHXI4PEZDY3I4AFZJY223MUJ2HRH7A2OA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:54.110997712 (541h25m53.982080757s) [noise=true , address=145.79.227.37:4566 , piece_id=WULR2SAELC7UFQWDRHTQ7RD4QV2TWF2IWV6JR3LRJUCOJPDJLMPA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:54.634168397 (541h25m54.505251432s) [noise=true , address=145.79.227.51:4410 , piece_id=JHAO2QMP6UU4GBSYEGZOB67AXMMYL33ISL57RQLK4I7CQ54FKTWA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:54.844988341 (541h25m54.716071386s) [noise=true , address=145.79.227.12:4765 , piece_id=QLSSBYEI76XZQHJBHMNFVRRRU4G234NZGSE23DZ3OORXEOEKGHGQ, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:54:58.331306556 (541h25m58.202389591s) [noise=true , address=145.79.227.29:4941 , piece_id=OKZVRGHADKX2TM6T7LX6WJM7NLKMRGYZLHAWAHU2HQMMMH2UOTMA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:55:07.710412450 (541h26m7.581495495s) [noise=true , address=145.79.227.10:4403 , piece_id=J4BAJTPKS4Q35DNDKQ3AKL3PCAQMNP3CV6DWVGYNK3TKISR3O52A, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:55:15.123476617 (541h26m14.994559662s) [noise=true , address=145.79.227.48:4303 , piece_id=VRW6US5GNPTQAJP5CWPXRKUP234ULZ4SJHSJHILZYFZTKIM3KU6A, seg_pos={0 0}, upload=474812]: Done uploading piece. err: <nil> Dec 28 04:55:18.971916354 (541h26m18.842999389s) [noise=true , address=145.79.227.52:4447 , piece_id=EVLCICHGDCSN5GNKMNVNGEVD4PY6QOU2B6D7I5EH6EC5AOG2XWZA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:145.79.227.52:4447): protocol: read tcp 188.166.225.170:41730->145.79.227.52:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:55:18.972105415 (541h26m18.84318846s) [noise=true , address=145.79.227.45:4231 , piece_id=NG7VBSJ56VPOKU7ND54LHLKY6NZSSX6KXQUKPM2FUI4Q7MH3S34Q, seg_pos={0 0}, upload=474812]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:145.79.227.45:4231): protocol: read tcp 188.166.225.170:52538->145.79.227.45:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 04:55:18.972198081 (541h26m18.843281256s) [noise=true , address=145.79.227.46:4601 , piece_id=FFE47R5Z57XU56I2CE7DOO4DGJLEZ3T2GMSQA65UT67BU6IPBH3Q, seg_pos={0 0}, upload=474812]: Done uploading piece. err: ecclient: upload failed (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5, address:145.79.227.46:4601): protocol: expected piece hash; context deadline exceeded; EOF Dec 28 04:55:18.972615295 (541h26m18.84369833s) [noise=true , address=145.79.227.38:4977 , piece_id=VGWXTQUT3M5JJDTKJRQBXEDDQCBH5LL6SAVBBQOBADDXEKJHVKLA, seg_pos={0 0}, upload=474812]: Done uploading piece. err: ecclient: upload failed (node:125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, address:145.79.227.38:4977): protocol: context deadline exceeded; context deadline exceeded Dec 28 04:55:18.972682253 (541h26m18.843765288s) [seg_pos={0 0}, upload=474812]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 28 04:55:19.220591810 (541h26m19.091674845s) [upload=474812]: Done uploading segments... Dec 28 04:55:19.220624301 (541h26m19.091707347s) [upload=474812]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.250531239s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.005959058s trace: https://jaeger.dev.storj.io/trace/3caebe97a0f0590e FAIL (2025-12-28 04:55:13 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 30.005979526s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......X................................................] duration: 42.010403922s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:54:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 42.010423951s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [....X.X...XXXXX....XX..X.....XXX.XX.......X..X.XXXX.....XX..] duration: 3.943653218s trace: https://jaeger.dev.storj.io/trace/6bb9ab3472ef1784 FAIL (2025-12-28 04:54:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.943681791s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.XXXXXX..XX..XX...X.....X..XXXXX..XX..X.XXX.X.X..X...X..XXX.] duration: 6.439570213s trace: https://jaeger.dev.storj.io/trace/701d7ac209094715 FAIL (2025-12-28 04:54:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 04:52:57.918408250 (541h23m57.789491285s) [upload=474784]: Starting upload Dec 28 04:52:57.918473895 (541h23m57.789556931s) [upload=474784]: Done starting upload Dec 28 04:52:57.918783044 (541h23m57.789866089s) [upload=474784]: Uploading segments... Dec 28 04:52:57.918869348 (541h23m57.789952383s) [upload=474784]: Got next segment. Inline: false Dec 28 04:52:57.918900868 (541h23m57.789983913s) [upload=474784]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 04:52:57.918905397 (541h23m57.789988432s) [upload=474784]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 04:52:58.165602625 (541h23m58.03668567s) [seg_pos={0 0}, upload=474784]: Begin upload segment... Dec 28 04:52:58.165699410 (541h23m58.036782455s) [seg_pos={0 0}, upload=474784]: Done begin upload segment. Dec 28 04:52:58.165745998 (541h23m58.036829083s) [noise=true , address=145.79.227.15:4874 , piece_id=CVQXNQVDLL377EH43H2XNQVUJWO3QM4XCBURHVWFA5NGU7NTQUIQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.165913176 (541h23m58.036996222s) [noise=true , address=45.91.116.10:4082 , piece_id=DNDRI6R325BARR3OW67PEWSCUGYPH3VMHCQ45QPDYZPXL7WEZPIQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.166774156 (541h23m58.037857191s) [noise=true , address=145.79.227.5:4222 , piece_id=4GQO4AW2HW5TAETKCZLR4P6ZEPQLEA4YQOXOQWDCPFJ4ESB4MP3A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.166816276 (541h23m58.037899311s) [noise=true , address=145.79.227.35:4482 , piece_id=VJKDZMJS3KRUMOM6BDX545FXB5GM4F65YKVUSLKVQ35ZGGQ4FPYA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167248920 (541h23m58.038331985s) [noise=true , address=145.79.227.6:4257 , piece_id=FRFKUD7RZXCEMMA73TFBYMU7QCQWNGBUPVUY6HK23AYCQZ6JHUOQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167392754 (541h23m58.038475789s) [noise=true , address=77.90.31.14:4663 , piece_id=5JQ465AGLMD457OLND4O3XI6NMIYPGNT7KJPAMK47VYQ2Y53VKYQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167617742 (541h23m58.038700787s) [noise=true , address=145.79.227.36:4519 , piece_id=ER4VFPHO3QUAM3JZPU4ZULUM2RRO2HU3YSGVABNVDKQU6T2KK5EQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167955335 (541h23m58.03903839s) [noise=true , address=77.90.31.19:4844 , piece_id=WNDL74JK4RATDVSGHJWHFLJD4E4LSRI3V7G7YYSAVRCZFSDDEV6A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.167969201 (541h23m58.039052236s) [noise=true , address=145.79.227.39:4015 , piece_id=GOQENX5CEY3TKIHJYDZJTNZ6VIRMX4B3LRXYVOMEDPKPQJ6VXFEA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168239336 (541h23m58.039322371s) [noise=true , address=145.79.227.37:4567 , piece_id=Q56OJOMH2WZLGVWYAKXG45QF4HQXSTFQNYM3VF5W46LQR2E6Y5QA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168571358 (541h23m58.039654393s) [noise=true , address=45.91.116.4:4937 , piece_id=MN43HKOVCGZTY2EM2QCLSFP4LO4S7TMYJ7PLBNANMMZON7WZNC2A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168598650 (541h23m58.039681695s) [noise=true , address=145.79.227.42:4122 , piece_id=MWMYEVTN6YKU36IWKQ7TVIH25BN4CMMZGMMYI3I4OTWUZOYHEWBA, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.168841212 (541h23m58.039924247s) [noise=true , address=77.90.31.12:4590 , piece_id=GW4AODTA2G2ULV6DTFBQKHOF5E2EMYZYSCTVLYD5HJDTWVFBMB5Q, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.169174677 (541h23m58.040257712s) [noise=true , address=45.91.116.12:4227 , piece_id=4S6RWT7NJVYU7OIGIC7MHOWUDOZQB4CCCCCLIRRZQI5BKMI7VWWQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.169827650 (541h23m58.040910695s) [noise=true , address=145.79.227.13:4802 , piece_id=TPGHRJTLR3TETNDIEDFGQ76FGPXNN4DE5JOPJDMV645QK5SFEB6A, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:58.170489139 (541h23m58.041572174s) [noise=true , address=77.90.31.21:4877 , piece_id=RSRSE2P6UKF3BIDZK7DIGWG6ZPVTDZZEYHW5A5F6D6SWBFUJX7EQ, seg_pos={0 0}, upload=474784]: Uploading piece... Dec 28 04:52:59.885866567 (541h23m59.756949602s) [upload=474784]: Next returned nil segment Dec 28 04:52:59.885889631 (541h23m59.756972686s) [upload=474784]: Waiting for error group managing segments... Dec 28 04:53:00.185504883 (541h24m0.056587929s) [noise=true , address=145.79.227.15:4874 , piece_id=CVQXNQVDLL377EH43H2XNQVUJWO3QM4XCBURHVWFA5NGU7NTQUIQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.185687903 (541h24m0.056770938s) [noise=true , address=145.79.227.39:4015 , piece_id=GOQENX5CEY3TKIHJYDZJTNZ6VIRMX4B3LRXYVOMEDPKPQJ6VXFEA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.189399379 (541h24m0.060482414s) [noise=true , address=145.79.227.42:4122 , piece_id=MWMYEVTN6YKU36IWKQ7TVIH25BN4CMMZGMMYI3I4OTWUZOYHEWBA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.190123118 (541h24m0.061206153s) [noise=true , address=145.79.227.35:4482 , piece_id=VJKDZMJS3KRUMOM6BDX545FXB5GM4F65YKVUSLKVQ35ZGGQ4FPYA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.202143737 (541h24m0.073226772s) [noise=true , address=145.79.227.13:4802 , piece_id=TPGHRJTLR3TETNDIEDFGQ76FGPXNN4DE5JOPJDMV645QK5SFEB6A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.202523209 (541h24m0.073606254s) [noise=true , address=145.79.227.37:4567 , piece_id=Q56OJOMH2WZLGVWYAKXG45QF4HQXSTFQNYM3VF5W46LQR2E6Y5QA, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.211302837 (541h24m0.082385872s) [noise=true , address=145.79.227.36:4519 , piece_id=ER4VFPHO3QUAM3JZPU4ZULUM2RRO2HU3YSGVABNVDKQU6T2KK5EQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.217272053 (541h24m0.088355098s) [noise=true , address=145.79.227.6:4257 , piece_id=FRFKUD7RZXCEMMA73TFBYMU7QCQWNGBUPVUY6HK23AYCQZ6JHUOQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.433488362 (541h24m0.304571407s) [noise=true , address=45.91.116.10:4082 , piece_id=DNDRI6R325BARR3OW67PEWSCUGYPH3VMHCQ45QPDYZPXL7WEZPIQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.468101320 (541h24m0.339184365s) [noise=true , address=77.90.31.21:4877 , piece_id=RSRSE2P6UKF3BIDZK7DIGWG6ZPVTDZZEYHW5A5F6D6SWBFUJX7EQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.490711994 (541h24m0.361795039s) [noise=true , address=77.90.31.19:4844 , piece_id=WNDL74JK4RATDVSGHJWHFLJD4E4LSRI3V7G7YYSAVRCZFSDDEV6A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.556360428 (541h24m0.427443473s) [noise=true , address=145.79.227.5:4222 , piece_id=4GQO4AW2HW5TAETKCZLR4P6ZEPQLEA4YQOXOQWDCPFJ4ESB4MP3A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.570783703 (541h24m0.441866738s) [noise=true , address=45.91.116.4:4937 , piece_id=MN43HKOVCGZTY2EM2QCLSFP4LO4S7TMYJ7PLBNANMMZON7WZNC2A, seg_pos={0 0}, upload=474784]: Done uploading piece. err: <nil> Dec 28 04:53:00.570803741 (541h24m0.441886776s) [seg_pos={0 0}, upload=474784]: Segment reached optimal threshold of 13 pieces. Dec 28 04:53:00.570993462 (541h24m0.442076538s) [noise=true , address=77.90.31.14:4663 , piece_id=5JQ465AGLMD457OLND4O3XI6NMIYPGNT7KJPAMK47VYQ2Y53VKYQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 28 04:53:00.571003722 (541h24m0.442086787s) [noise=true , address=77.90.31.14:4663 , piece_id=5JQ465AGLMD457OLND4O3XI6NMIYPGNT7KJPAMK47VYQ2Y53VKYQ, seg_pos={0 0}, upload=474784]: Long tail cancellation due to optimal threshold Dec 28 04:53:00.571152184 (541h24m0.442235219s) [noise=true , address=45.91.116.12:4227 , piece_id=4S6RWT7NJVYU7OIGIC7MHOWUDOZQB4CCCCCLIRRZQI5BKMI7VWWQ, seg_pos={0 0}, upload=474784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:45178->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:45178->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45178->45.91.116.12:4227: use of closed network connection Dec 28 04:53:00.571158215 (541h24m0.44224125s) [noise=true , address=45.91.116.12:4227 , piece_id=4S6RWT7NJVYU7OIGIC7MHOWUDOZQB4CCCCCLIRRZQI5BKMI7VWWQ, seg_pos={0 0}, upload=474784]: Long tail cancellation due to optimal threshold Dec 28 04:53:00.571485469 (541h24m0.442568524s) [noise=true , address=77.90.31.12:4590 , piece_id=GW4AODTA2G2ULV6DTFBQKHOF5E2EMYZYSCTVLYD5HJDTWVFBMB5Q, seg_pos={0 0}, upload=474784]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 04:53:00.571506499 (541h24m0.442589544s) [noise=true , address=77.90.31.12:4590 , piece_id=GW4AODTA2G2ULV6DTFBQKHOF5E2EMYZYSCTVLYD5HJDTWVFBMB5Q, seg_pos={0 0}, upload=474784]: Long tail cancellation due to optimal threshold Dec 28 04:53:00.571576261 (541h24m0.442659307s) [seg_pos={0 0}, upload=474784]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 04:53:00.571688685 (541h24m0.44277173s) [upload=474784]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:53:00.571697673 (541h24m0.442780708s) [upload=474784]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:53:00.861235446 (541h24m0.732318491s) [upload=474784]: Done uploading segments... Dec 28 04:53:00.861273549 (541h24m0.732356594s) [upload=474784]: Upload finished. err: <nil> Duration: 6.439637372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X......X...............................................] duration: 59.051729003s trace: https://jaeger.dev.storj.io/trace/26c246d73008cc49 FAIL (2025-12-28 04:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179395486 (540h42m11.050478531s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.179823281 (540h42m11.050906336s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180232490 (540h42m11.051315535s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.180945087 (540h42m11.052028132s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181189232 (540h42m11.052272277s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181508139 (540h42m11.052591174s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181597209 (540h42m11.052680254s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.181882131 (540h42m11.052965166s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182132017 (540h42m11.053215052s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182274007 (540h42m11.053357052s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182451545 (540h42m11.05353458s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182777516 (540h42m11.053860551s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.182919286 (540h42m11.054002331s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183101804 (540h42m11.054184839s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:11.183514740 (540h42m11.054597785s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Uploading piece... Dec 28 04:11:12.562530634 (540h42m12.433613679s) [upload=474194]: Next returned nil segment Dec 28 04:11:12.562544100 (540h42m12.433627135s) [upload=474194]: Waiting for error group managing segments... Dec 28 04:11:13.045899872 (540h42m12.916982917s) [noise=true , address=145.79.227.9:4371 , piece_id=2ON3P6ZFEUEBFFLCHYY5BWAAJEFX43KKU5MU5TLBQHSZ52RCAYLA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.068300556 (540h42m12.939383591s) [noise=true , address=145.79.227.11:4755 , piece_id=F37J7GC6RB2PL3B4B6MFVAPB6IOGSJSKVQ5UNFHFSBJ7C3SABURQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.073683536 (540h42m12.944766571s) [noise=true , address=145.79.227.40:4053 , piece_id=V7R5MTCCKCGHV7KD4LAXFBUJI3HRGVAJUCVYL4WPL3M32CVQXOBA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.125067800 (540h42m12.996150835s) [noise=true , address=145.79.227.6:4267 , piece_id=SDG3WH6TPR47OHG57NMVFAHOYVAL45XMJ7S22FZL6IFR6DS45HQQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.156395944 (540h42m13.027478989s) [noise=true , address=145.79.227.51:4432 , piece_id=C5P63KNNZLSOBH772MRMYOJOQGLE7KXGELQN437HTVPAM72RXUCA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.221620070 (540h42m13.092703105s) [noise=true , address=77.90.31.15:4714 , piece_id=YVXSR2B74BDK4BV5QGLZLWOKVXV3TWKSW5M3KG323U7VUUBIQU3Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.266169559 (540h42m13.137252594s) [noise=true , address=145.79.227.33:4876 , piece_id=UEOY3HYI3WWJTXUSBALBXIGCMBC5M6XGR42SIOBXQNDCZ3UFMWVA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.273253176 (540h42m13.144336221s) [noise=true , address=77.90.31.12:4604 , piece_id=DVQ53JEZL2GDTXFDDIWN4BLSJJHRZORII7UNF5HS3GIM5VJX3Y2Q, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.324930308 (540h42m13.196013343s) [noise=true , address=77.90.31.17:4778 , piece_id=WHTKIVKEZNFREOTNDTLLBB24JGWKIXWBJKZ2EUM3QL2SM7Z4YXDA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.333001758 (540h42m13.204084793s) [noise=true , address=77.90.31.18:4819 , piece_id=FVZE5P55HHT3RGOI4M2OFV6NER4BNVGFSZOMDRADO5JYI54DRHLQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.418239402 (540h42m13.289322437s) [noise=true , address=45.91.116.6:4003 , piece_id=H4VWZIMYVM2M3SX76LSS64ARSENBQL6OOTL5EY5XE4IAORO7OWGQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.460099681 (540h42m13.331182726s) [noise=true , address=77.90.31.21:4906 , piece_id=SGYCZ2E6IMEJA332PTI7DQ4FWRJR6NG4OYRUP6H2KHXS2TKC5S2A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:13.860538777 (540h42m13.731621812s) [noise=true , address=193.149.164.143:4696 , piece_id=OJQFP4YJ3EF52S2XQN5QPKMQJDS57QEOQQ4PUPARPGPAVSLGKMFA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127606583 (540h42m13.998689618s) [noise=true , address=145.79.227.34:4924 , piece_id=R7MT4YS5IPGHOCNZX77ITOPXL57DOYD2HJXPJYYBGNVUWOT33JPA, seg_pos={0 0}, upload=474194]: Done uploading piece. err: <nil> Dec 28 04:11:14.127625258 (540h42m13.998708303s) [seg_pos={0 0}, upload=474194]: Segment reached optimal threshold of 14 pieces. Dec 28 04:11:14.127786726 (540h42m13.998869761s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DSHTtpwcx1h1utGXNou11K7Tcg1ZhkkMdMgY6BDD2ifG6LQ7S): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.127792547 (540h42m13.998875582s) [noise=true , address=45.91.116.4:4966 , piece_id=ZTHC25VIBEHSKRULTNAS25DK2DBVI7ATO46SDMMTON7PBBPRMWBQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128017335 (540h42m13.99910037s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128023747 (540h42m13.999106782s) [noise=true , address=145.79.227.43:4191 , piece_id=CLYBACZJDTAFTBEMEMSOXP4I22PDYSVIBX3AMNN4DISQUJWYFDMQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128139027 (540h42m13.999222102s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40936->64.49.13.169:4708: use of closed network connection Dec 28 04:11:14.128150668 (540h42m13.999233733s) [noise=true , address=64.49.13.169:4708 , piece_id=6OBO4M26AXDJXUTOBI66S2JJARDEM2CTTFHUIAOI4CX2PZITRWFQ, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128292198 (540h42m13.999375243s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pfXWex2WsuhmLb2smRpE46cHT9aipSTHW3e5oyELS1GhF1cFr): protocol: expected piece hash; context canceled; EOF Dec 28 04:11:14.128299212 (540h42m13.999382257s) [noise=true , address=145.79.227.30:4784 , piece_id=UTH47JZWG3SGX3KXBFJJALALTPP6INYORUHDYSS6CNDEUQ6BDL7A, seg_pos={0 0}, upload=474194]: Long tail cancellation due to optimal threshold Dec 28 04:11:14.128395815 (540h42m13.99947885s) [seg_pos={0 0}, upload=474194]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 04:11:14.128488171 (540h42m13.999571206s) [upload=474194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 04:11:14.128493561 (540h42m13.999576597s) [upload=474194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 04:11:14.572773387 (540h42m14.443856452s) [upload=474194]: Done uploading segments... Dec 28 04:11:14.572800389 (540h42m14.443883454s) [upload=474194]: Upload finished. err: <nil> Duration: 59.051757546s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)