Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.644471785s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:30:25 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.644501733s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 3.247140704s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:30:23 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.24717489s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.718519034s trace: https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24 FAIL (2025-12-23 02:30:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments... Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments... Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.718552409s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.589551587s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:30:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.589581994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.821674727s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:30:09 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.821696106s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.193534451s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:30:07 UTC) orbiter warning: context deadline exceeded Duration: 6.193568516s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.802411214s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:30:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 5.802441914s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.559711246s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:29:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559740411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.823902347s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:29:52 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.8239304s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 4.941135238s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:29:52 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 4.941163182s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.930065558s trace: https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24 FAIL (2025-12-23 02:29:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments... Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments... Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.930089763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.870181328s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:29:35 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.870308159s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.71629135s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:29:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 5.716319824s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.484834111s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:29:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 6.484889959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 3.311582248s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:29:15 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.311608808s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 14.957256329s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:29:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 14.957286648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.287751981s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:29:02 UTC) orbiter warning: context deadline exceeded Duration: 6.287769826s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.029542545s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:28:59 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 6.029568435s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 3.351172557s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:28:57 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.351189259s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 7.608760223s trace: https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24 FAIL (2025-12-23 02:28:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments... Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments... Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 7.608787285s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.29564212s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:28:46 UTC) orbiter warning: context deadline exceeded Duration: 6.295759945s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.619809281s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:28:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.619865537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 4.943183409s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:28:41 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 4.943213777s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 9.023952597s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:28:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 9.023997512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.772990273s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:28:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 5.773012805s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.783714554s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:28:20 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.783746574s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.77864217s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:28:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 6.778669854s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.005396173s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:28:07 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 5.005423354s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.235580272s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:27:59 UTC) orbiter warning: context deadline exceeded Duration: 6.235692857s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.502663507s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:27:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502692002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 14.832402405s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:27:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 14.832424979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.553512603s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:27:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 6.553549464s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 3.240874995s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:27:42 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.240905676s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.69111049s trace: https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24 FAIL (2025-12-23 02:27:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments... Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments... Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 5.691155916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.956087537s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:27:36 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.956110781s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.148916432s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:27:36 UTC) orbiter warning: context deadline exceeded Duration: 6.148942932s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.008598809s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:27:31 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 5.008626281s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 18.174557716s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:27:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 18.174585439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.958320886s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:27:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 5.958348066s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.533071837s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:27:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533097587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.358185493s trace: https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24 FAIL (2025-12-23 02:26:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments... Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments... Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 6.358215139s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.341614511s trace: https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca FAIL (2025-12-23 02:26:51 UTC) orbiter warning: context deadline exceeded Duration: 6.341641493s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.897594646s trace: https://jaeger.dev.storj.io/trace/70da446b2b2ff016 FAIL (2025-12-23 02:26:40 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46) Duration: 5.897620105s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 3.431387334s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:26:38 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.431415288s core uplink ap1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 4.963463088s trace: https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9 FAIL (2025-12-23 02:26:32 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45) Duration: 4.963488054s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 7.12485844s trace: https://jaeger.dev.storj.io/trace/463d97d26edafa8a FAIL (2025-12-23 02:26:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=367700]: Uploading piece... Dec 23 02:17:52.581508683 (418h48m52.452591718s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581711800 (418h48m52.452794835s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.581958349 (418h48m52.453041384s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582064401 (418h48m52.453147436s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582336870 (418h48m52.453419905s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582391083 (418h48m52.453474118s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582715682 (418h48m52.453798717s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.582847693 (418h48m52.453930738s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583274976 (418h48m52.454358022s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:52.583662655 (418h48m52.45474569s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Uploading piece... Dec 23 02:17:53.405090439 (418h48m53.276173484s) [upload=367700]: Next returned nil segment Dec 23 02:17:53.405105007 (418h48m53.276188042s) [upload=367700]: Waiting for error group managing segments... Dec 23 02:17:53.798794305 (418h48m53.66987735s) [noise=true , address=185.167.204.205:4025 , piece_id=XY5XNAAQZFPKWJP7V5LWBBH5ZQ7XFUTQ657QEF5KXFTP2EKCSIBA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799500561 (418h48m53.670583596s) [noise=true , address=185.167.204.205:4016 , piece_id=523RGOAK7OI6VYQY5X4XT5Y6F5C7NMB2WDXPN74Q27P3BCH2Q37A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.799686735 (418h48m53.67076977s) [noise=true , address=185.167.204.205:4036 , piece_id=IQ6I5X4HC2ANQSM72N7X4OJWVTUMOWH5YPSSRHUFTGYSXW5ZK4WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.800348164 (418h48m53.671431209s) [noise=true , address=185.167.204.205:4033 , piece_id=42CCWQIIXKFWWCLUSLGSFQHZMA2UPGZJMOSWD7Z562DV6ABPDVRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802008737 (418h48m53.673091782s) [noise=true , address=185.167.204.206:4057 , piece_id=UNUF4NZ5CL74EWLPZLVX44MAYYUQFIAZ54N7FGBLCQHEKLGX6RSQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.802484733 (418h48m53.673567778s) [noise=true , address=185.167.204.206:4040 , piece_id=Y6HVWGUESE2BSEIKEMDSLNHWDTOEKGBR76NIFRRQU6KFTVHD4Q7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803136594 (418h48m53.674219639s) [noise=true , address=185.167.204.206:4070 , piece_id=GA3GZFHS36RXW4PWKSHZ2QOAXQJVNIWVMED3XAAGG5GQPA27ST7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.803750904 (418h48m53.674833939s) [noise=true , address=185.167.204.206:4053 , piece_id=GG5TA7D4KK2IGSKRO4T3APKSZ43IWLND3L54EXYD2LRFIKGI5MRQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.804347520 (418h48m53.675430565s) [noise=true , address=185.167.204.206:4058 , piece_id=4CZUKD7KF6H3JH6DLWT3KM2PQC72NRYW4YPN7NL445VBER4QR6SA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805167522 (418h48m53.676250567s) [noise=true , address=185.167.204.206:4054 , piece_id=3QWOKGEC5JODMZCL5R5ULU7EUAKGI2W5QWONT5H2I7EOBH7PVODA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.805898293 (418h48m53.676981328s) [noise=true , address=185.167.204.206:4066 , piece_id=WAVNL7LBCOQEW3CFA3DWRITRTBGCWYVASH43KC4RFHE5KFQUQ45A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808200908 (418h48m53.679283943s) [noise=true , address=185.167.204.206:4064 , piece_id=4K4KD2ZMFXUZT6W3OV7E6BC5UGQHSVZXR67B2QPXIPKGAM6CTVSA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808393705 (418h48m53.67947675s) [noise=true , address=185.167.204.205:4004 , piece_id=G5Q7QERF2O3WXIDK5ZAFGSZIN6VXJOFYHAPEC647NIXIJQCSNLKA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.808844694 (418h48m53.679927739s) [noise=true , address=185.167.204.205:4028 , piece_id=DINID3XD5734Z3RWWS3WPIFLGRUXZMZUNXC2BC3NMX7LWTSDVN3Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.810924914 (418h48m53.682007959s) [noise=true , address=185.167.204.206:4047 , piece_id=JSWKSSXVNEFRZPWP56EHG26DRYA7V4K2LPEBCKUOBUG2WET5ZQ7A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813330996 (418h48m53.684414031s) [noise=true , address=185.167.204.206:4069 , piece_id=OLTP357X7TH2ABIRWBMAI445KR5SGJVY77PPIQFLRD2KFCBOML2Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.813884790 (418h48m53.684967826s) [noise=true , address=185.167.204.205:4001 , piece_id=LMIOF4TXW2JJBYGBNU36M46PXZUNKBKFK56XVNMWRQVR7LSZ3QCQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:53.815627059 (418h48m53.686710094s) [noise=true , address=185.167.204.206:4067 , piece_id=D24D7H4IJKMKHRJUCQ2ZAYDIVRUYHK2YHEYLVCIL3VA4P5FQJB5A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.448896803 (418h48m54.319979838s) [noise=true , address=81.171.14.70:4029 , piece_id=JKKAJ3PXL7WDZ7XNKM752C6NA4B4FUN2AZJCGAZPDVNAGKTAOK7Q, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.835287845 (418h48m54.70637088s) [noise=true , address=81.171.14.70:4025 , piece_id=PFRRLSNJYBUCYSOVOE3VHDWAKTEPMPQRTE4U2VTPBT7KGYVNQHIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.946890399 (418h48m54.817973434s) [noise=true , address=81.171.14.70:4040 , piece_id=6ZIPFOQCGRNK3TDTODGI34TL2WM65BQSRLXXAFX3XQ3VCPAVT2YA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:54.960812961 (418h48m54.831895996s) [noise=true , address=81.171.14.70:4034 , piece_id=2EA6SNLDV3ZVTNQ5UXEXIC2SJTPH3D6VCVZJI2IAGS22U2J6Y6WQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.069208920 (418h48m54.940291965s) [noise=true , address=89.149.201.147:4009 , piece_id=WGVNGAHNZIVLBR665VB7GO77EZ7SSJOC5P7ML2UAK6MKV4BAF27A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.073463732 (418h48m54.944546768s) [noise=true , address=89.149.201.147:4002 , piece_id=QCDLIQPY7COKADVRRRUN7UBEIV2AARIPA64LMFCCYPFNA54TA7VQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.132812454 (418h48m55.003895489s) [noise=true , address=89.149.201.147:4010 , piece_id=LZXMVADI5CHNTPWZNXNBEFI3SMGAWQGNUIWDNPM4JMRBLZ62HO4A, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185887030 (418h48m55.056970065s) [noise=true , address=89.149.201.147:4012 , piece_id=2Z64OIS2BBYH6ZCKBXHW4XQZY4FFASHDIHH3C3YVY7MGTTBI6QNA, seg_pos={0 0}, upload=367700]: Done uploading piece. err: <nil> Dec 23 02:17:55.185911748 (418h48m55.056994783s) [seg_pos={0 0}, upload=367700]: Segment reached optimal threshold of 26 pieces. Dec 23 02:17:55.186137848 (418h48m55.057220893s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RbD6M2MHJfxcJXogZG8nUGFmeZjwUQ1auN6KGWJMPTffdK42K): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186148097 (418h48m55.057231132s) [noise=true , address=89.149.201.147:4015 , piece_id=WENNEH2N2S7NV3TQILPUZLFRBV5CO6VMP52NKDIQE23NO2X7GDWQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186212209 (418h48m55.057295244s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XsvrVhDGkU8BkYsBGmTtax47E43dZAYuGwD8WBMpp3n4GgDM5): protocol: expected piece hash; context canceled; EOF Dec 23 02:17:55.186228150 (418h48m55.057311185s) [noise=true , address=89.149.201.147:4016 , piece_id=MFNURTZ2HJFACIAHZUO7YZJXRJZB26AVF24JLU3I7IYKDPZ2JJIQ, seg_pos={0 0}, upload=367700]: Long tail cancellation due to optimal threshold Dec 23 02:17:55.186247767 (418h48m55.057330802s) [seg_pos={0 0}, upload=367700]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 23 02:17:55.186337368 (418h48m55.057420403s) [upload=367700]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:17:55.186342387 (418h48m55.057425422s) [upload=367700]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:17:55.430535735 (418h48m55.30161878s) [upload=367700]: Done uploading segments... Dec 23 02:17:55.430557656 (418h48m55.301640701s) [upload=367700]: Upload finished. err: <nil> Duration: 7.124888368s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 9.507552188s trace: https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7 FAIL (2025-12-23 02:26:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece... Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments... Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil> Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces. Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments... Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil> Duration: 9.507580633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 3.165052105s trace: https://jaeger.dev.storj.io/trace/20c185d15563c01c FAIL (2025-12-23 02:26:16 UTC) orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39) Duration: 3.165079356s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.114745947s trace: https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24 FAIL (2025-12-23 02:25:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 2) ======LOGS====== 7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece... Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments... Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil> Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil> Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments... Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2) Duration: 6.114775262s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.566825247s trace: https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793 FAIL (2025-12-23 02:25:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566910199s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)