Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X.XX....XXX..X..X......XX..............X.................] duration: 8.190471438s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:06:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 8.190501414s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 21.797729243s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:06:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 21.797773969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X.XX....XXX..X..X......XX..............X................] duration: 6.030223896s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:05:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 6.030252511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 15.14375138s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:04:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.143784813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X.XX....XXX..X..X......XX..............X...............] duration: 15.371772322s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:04:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 15.371803443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....X.XX....XXX..X..X......XX..............X..............] duration: 6.028958927s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:03:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 6.028990827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 15.380595873s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:03:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.380609459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 25.206608624s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:03:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 25.206629725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.....X.XX....XXX..X..X......XX..............X.............] duration: 7.820713482s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:02:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 7.820743128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 23.251508147s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:02:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 23.251538083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.....X.XX....XXX..X..X......XX..............X............] duration: 13.149609726s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:02:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 13.149642367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 8.388949707s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:02:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 8.388976919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.856702306s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:01:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 6.856742463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 9.681652328s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:01:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 9.681683517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.....X.XX....XXX..X..X......XX..............X...........] duration: 6.379131714s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 20:00:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 6.379216446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 15.401001066s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 20:00:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.401030481s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X.....X.XX....XXX..X..X......XX..............X..........] duration: 6.029664856s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:59:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 6.029698841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 20.312630227s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:59:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 20.312655745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.....X.XX....XXX..X..X......XX..............X.........] duration: 25.953032567s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:59:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 25.953060091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..............................X............................] duration: 15.369507735s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:59:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.369543893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..............................X...........................] duration: 15.150713889s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:58:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.150742614s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X.....X.XX....XXX..X..X......XX..............X........] duration: 17.154815528s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:58:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 17.154866486s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..............................X..........................] duration: 17.948120685s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:58:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 17.948150822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..............................X.........................] duration: 15.495833683s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:57:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.495865053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X......X.....X.XX....XXX..X..X......XX..............X.......] duration: 6.151658133s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:56:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 6.151705382s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..............................X........................] duration: 18.030241108s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:56:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 18.030266316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X......X.....X.XX....XXX..X..X......XX..............X......] duration: 15.030706641s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:56:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 15.030746026s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..............................X.......................] duration: 15.515074821s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:56:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.515104418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..............................X......................] duration: 17.948162651s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:55:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 17.94818929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X......X.....X.XX....XXX..X..X......XX..............X.....] duration: 5.991554428s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:55:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 5.991601958s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..............................X.....................] duration: 15.313108361s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:54:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.313142887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X......X.....X.XX....XXX..X..X......XX..............X....] duration: 15.811229925s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 15.811263219s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X......X.....X.XX....XXX..X..X......XX..............X...] duration: 25.732550396s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:53:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 25.732575463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..............................X....................] duration: 25.245100419s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:53:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 25.245131728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..............................X...................] duration: 17.859854138s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:53:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 17.859873405s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X......X.....X.XX....XXX..X..X......XX..............X..] duration: 5.760226101s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:52:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 5.760245277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..............................X..................] duration: 19.237340219s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:52:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 19.23736736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X......X.....X.XX....XXX..X..X......XX..............X.] duration: 6.073106586s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:52:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 6.073126142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X......X.....X.XX....XXX..X..X......XX..............X] duration: 30.00332981s trace: https://jaeger.dev.storj.io/trace/6de23ce395480be2 FAIL (2025-12-28 19:52:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA) context deadline exceeded ======LOGS====== Dec 28 19:51:31.138959699 (556h22m31.010042744s) [upload=487778]: Starting upload Dec 28 19:51:31.139044411 (556h22m31.010127456s) [upload=487778]: Done starting upload Dec 28 19:51:31.139127078 (556h22m31.010210113s) [upload=487778]: Uploading segments... Dec 28 19:51:31.139326739 (556h22m31.010409784s) [upload=487778]: Got next segment. Inline: false Dec 28 19:51:31.139377555 (556h22m31.0104606s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:51:31.139384197 (556h22m31.010467232s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:51:31.387475461 (556h22m31.258558496s) [seg_pos={0 0}, upload=487778]: Begin upload segment... Dec 28 19:51:31.387544984 (556h22m31.258628049s) [seg_pos={0 0}, upload=487778]: Done begin upload segment. Dec 28 19:51:31.387771626 (556h22m31.258854671s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.388814712 (556h22m31.259897747s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389204404 (556h22m31.260287449s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.389706850 (556h22m31.260789885s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390109167 (556h22m31.261192212s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390421501 (556h22m31.261504546s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.390852582 (556h22m31.261935617s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.392756678 (556h22m31.263839713s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393255988 (556h22m31.264339023s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.393633607 (556h22m31.264716642s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394230243 (556h22m31.265313278s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.394697414 (556h22m31.265780449s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.401284737 (556h22m31.272367792s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.402581947 (556h22m31.273664982s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.403710777 (556h22m31.274793822s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:31.404110447 (556h22m31.275193492s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Uploading piece... Dec 28 19:51:33.086369496 (556h22m32.957452531s) [upload=487778]: Next returned nil segment Dec 28 19:51:33.086389915 (556h22m32.95747296s) [upload=487778]: Waiting for error group managing segments... Dec 28 19:51:33.394411997 (556h22m33.265495032s) [noise=true , address=145.79.227.21:4475 , piece_id=BBVRNDFL25BI3Q2CNE36E5GRAOXT55T4MQA2HTXX257HPKZ6HSPQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.398162920 (556h22m33.269245955s) [noise=true , address=145.79.227.39:4013 , piece_id=PWYMH2SD6JOHCSM7HFDM4CHUD5C3MUCSCRML3BMZKHDZAFRM6UMA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.403651411 (556h22m33.274734446s) [noise=true , address=145.79.227.45:4229 , piece_id=AZOSHGLV2UUTXSX5NX27R3ZFIEOWS2CDGLSG34JZKCVWY4DLHUHQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.406027926 (556h22m33.277110971s) [noise=true , address=145.79.227.51:4411 , piece_id=LICH4YP46TD6FZHHN4X4OFWKYBPPCFWSGHU63E2WP4AMD6HFZDNA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.418392551 (556h22m33.289475586s) [noise=true , address=145.79.227.47:4265 , piece_id=43D7BZZXISPRFCKYXSTCZONLHLQ2Y6IRLFD7PDEUAJWYP6ID7DMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.420875018 (556h22m33.291958063s) [noise=true , address=145.79.227.41:4087 , piece_id=SILOBZTKKG4QOW6F2XRCEXG64GCTMC2ULONWTO2D3QFFL74FP5BQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.421544332 (556h22m33.292627377s) [noise=true , address=145.79.227.2:4113 , piece_id=QJTHNBJ6SMI7GHV5634NHXQUD2P4TXERE77FO4ECEVYHQ6GUQ3FQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.427849308 (556h22m33.298932353s) [noise=true , address=145.79.227.9:4366 , piece_id=PC3TE34FAHG6J6TWTEU7GKWFGP4DDUELNIT2RIAEKGTM6XDRWIZA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.649006914 (556h22m33.520089949s) [noise=true , address=45.91.116.9:4261 , piece_id=BQE24O2CURUFHQS4WSIG722BGBPCPOY32W6KAOECGV7JPCKWGOKA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.688630355 (556h22m33.55971339s) [noise=true , address=77.90.31.14:4661 , piece_id=XLUH6WZPNM2W67MLQ7KFKGM3Y4WL4TFJ6GSYKPX325FSDPKZCUUA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.694890445 (556h22m33.56597348s) [noise=true , address=45.91.116.5:4300 , piece_id=NYOXGWDJWHYHW6BY3DQHFGISYSYQV4JT2H2XDJ4WLMUNE2IHN6LA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:33.738432388 (556h22m33.609515433s) [noise=true , address=45.91.116.7:4011 , piece_id=NQDPXLIESHRAWWBPVYOGPARTQ6PDN3TO7YCWTLXI4KBFD7JZFXXQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314051275 (556h22m34.18513431s) [noise=true , address=77.90.31.10:4519 , piece_id=VL3CQZWQBCPGDPMEAGIQCCHVRPMV36BXHPHJ2LPJLIC6LDGWR5YA, seg_pos={0 0}, upload=487778]: Done uploading piece. err: <nil> Dec 28 19:51:34.314075711 (556h22m34.185158746s) [seg_pos={0 0}, upload=487778]: Segment reached optimal threshold of 13 pieces. Dec 28 19:51:34.314354963 (556h22m34.185437998s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314365733 (556h22m34.185448778s) [noise=true , address=45.91.116.6:4974 , piece_id=FSPEWZOEC4SHIHA6IJSXCHUCOL6GPFRDFUFXJPCQR7263Y2YFIMQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314429585 (556h22m34.18551263s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314451267 (556h22m34.185534302s) [noise=true , address=145.79.227.5:4221 , piece_id=O73W2GAUHPHLTWZTO7IJT4OBBXAX22MAABQ2MQSTZPNLPIQXHKRQ, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314521831 (556h22m34.185604866s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 28 19:51:34.314530057 (556h22m34.185613092s) [noise=true , address=77.90.31.16:4735 , piece_id=EBCUZV26NTWYQFRJEZQE3N5EU2FPNPTQRCEPR6FNK5YTRZRORP2Q, seg_pos={0 0}, upload=487778]: Long tail cancellation due to optimal threshold Dec 28 19:51:34.314593758 (556h22m34.185676793s) [seg_pos={0 0}, upload=487778]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:51:34.314673079 (556h22m34.185756124s) [upload=487778]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:51:34.314678910 (556h22m34.185761965s) [upload=487778]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:51:34.609917501 (556h22m34.481000566s) [upload=487778]: Done uploading segments... Dec 28 19:51:34.609974519 (556h22m34.481057594s) [upload=487778]: Upload finished. err: <nil> Duration: 30.003354846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..............................X.................] duration: 15.41804976s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:51:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 15.418094866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..............................X................] duration: 28.630479286s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:51:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 28.63049656s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X......X.....X.XX....XXX..X..X......XX..............] duration: 11.888349341s trace: https://jaeger.dev.storj.io/trace/59a4a4c6f8437ff6 FAIL (2025-12-28 19:50:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA) context deadline exceeded ======LOGS====== g segments... Dec 28 19:40:18.799221148 (556h11m18.670304183s) [upload=487612]: Got next segment. Inline: false Dec 28 19:40:18.799268729 (556h11m18.670351765s) [upload=487612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:40:18.799274560 (556h11m18.670357595s) [upload=487612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:40:19.046301604 (556h11m18.917384639s) [seg_pos={0 0}, upload=487612]: Begin upload segment... Dec 28 19:40:19.046366318 (556h11m18.917449363s) [seg_pos={0 0}, upload=487612]: Done begin upload segment. Dec 28 19:40:19.046427104 (556h11m18.917510139s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.046454065 (556h11m18.9175371s) [noise=true , address=145.79.227.6:4257 , piece_id=WAMRDM5M4SS3G75G25AYB2SENETYLZIMK4KNMA33UPXNNZMH4F2Q, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.047153687 (556h11m18.918236732s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.047726838 (556h11m18.918809873s) [noise=true , address=145.79.227.14:4838 , piece_id=XM5TUQ5HOKPBXW4TY4GS57UYQGXMRNYOFQUSOCPQLYRY7TB2J3KQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.048161617 (556h11m18.919244662s) [noise=true , address=145.79.227.15:4874 , piece_id=IQJQRSB5K2ICUNEWM3JYV5G7YKERKLKZAFXQ7ZIXJFU3ZPTZ6VPA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.048604028 (556h11m18.919687073s) [noise=true , address=145.79.227.13:4802 , piece_id=CMYYN6ANGBTFOSYKDUKPLHUBAGQ64EVK2ZCRCQL4GRFFI64QO4OA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.049957866 (556h11m18.921040911s) [noise=true , address=145.79.227.45:4229 , piece_id=N2ZOTUO37BFFTIE5MZQ45K7IJRAPDCEU5OSBYMAFZFPX5DI3C23Q, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.050387555 (556h11m18.9214706s) [noise=true , address=45.91.116.7:4009 , piece_id=GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.051621424 (556h11m18.922704459s) [noise=true , address=77.90.31.11:4555 , piece_id=KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.052026706 (556h11m18.923109741s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.052733111 (556h11m18.923816156s) [noise=true , address=145.79.227.24:4582 , piece_id=C2H5V4G5T3HPD3RL5BGL6IYWVPG4LS6JT5IEVKHXKC5IISBAN7RA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.053928258 (556h11m18.925011303s) [noise=true , address=145.79.227.21:4475 , piece_id=AWYU3CVGPDKUUANUYJZUKKB3GPLG7AOUTWX3X6TOHFSJNO6EOGMA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.055729047 (556h11m18.926812082s) [noise=true , address=45.91.116.10:4081 , piece_id=RKYX4LS7VSTTMSWMYAHBR7VZ5NQNQXSBT37337PZ3FF2OPYFDJWA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.056840893 (556h11m18.927923928s) [noise=true , address=45.91.116.12:4228 , piece_id=J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.058069784 (556h11m18.929152819s) [noise=true , address=145.79.227.37:4566 , piece_id=UT3U3SVVNTDS7REKJ4J4WYXYAZNS35YTOEYPU4V65DQW3AIX2RZA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.059195617 (556h11m18.930278652s) [noise=true , address=145.79.227.28:4727 , piece_id=G7THCZWS4A7PVWA4HI3WZGSFVQO4TZKC6BGNVEU4DQO5DMPCFPUQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:20.770961428 (556h11m20.642044463s) [upload=487612]: Next returned nil segment Dec 28 19:40:20.770980875 (556h11m20.64206392s) [upload=487612]: Waiting for error group managing segments... Dec 28 19:40:21.046457965 (556h11m20.917541s) [noise=true , address=145.79.227.15:4874 , piece_id=IQJQRSB5K2ICUNEWM3JYV5G7YKERKLKZAFXQ7ZIXJFU3ZPTZ6VPA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.059462639 (556h11m20.930545684s) [noise=true , address=145.79.227.24:4582 , piece_id=C2H5V4G5T3HPD3RL5BGL6IYWVPG4LS6JT5IEVKHXKC5IISBAN7RA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.061847620 (556h11m20.932930665s) [noise=true , address=145.79.227.21:4475 , piece_id=AWYU3CVGPDKUUANUYJZUKKB3GPLG7AOUTWX3X6TOHFSJNO6EOGMA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.063315404 (556h11m20.934398439s) [noise=true , address=145.79.227.14:4838 , piece_id=XM5TUQ5HOKPBXW4TY4GS57UYQGXMRNYOFQUSOCPQLYRY7TB2J3KQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.064388047 (556h11m20.935471082s) [noise=true , address=145.79.227.6:4257 , piece_id=WAMRDM5M4SS3G75G25AYB2SENETYLZIMK4KNMA33UPXNNZMH4F2Q, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.072512037 (556h11m20.943595072s) [noise=true , address=145.79.227.45:4229 , piece_id=N2ZOTUO37BFFTIE5MZQ45K7IJRAPDCEU5OSBYMAFZFPX5DI3C23Q, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.089000846 (556h11m20.960083881s) [noise=true , address=145.79.227.37:4566 , piece_id=UT3U3SVVNTDS7REKJ4J4WYXYAZNS35YTOEYPU4V65DQW3AIX2RZA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.156103774 (556h11m21.027186809s) [noise=true , address=145.79.227.13:4802 , piece_id=CMYYN6ANGBTFOSYKDUKPLHUBAGQ64EVK2ZCRCQL4GRFFI64QO4OA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.220520528 (556h11m21.091603563s) [noise=true , address=145.79.227.28:4727 , piece_id=G7THCZWS4A7PVWA4HI3WZGSFVQO4TZKC6BGNVEU4DQO5DMPCFPUQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.243044178 (556h11m21.114127223s) [noise=true , address=77.90.31.11:4555 , piece_id=KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.301399419 (556h11m21.172482464s) [noise=true , address=45.91.116.12:4228 , piece_id=J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.309939952 (556h11m21.181022997s) [noise=true , address=45.91.116.7:4009 , piece_id=GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.376542859 (556h11m21.247625904s) [noise=true , address=45.91.116.10:4081 , piece_id=RKYX4LS7VSTTMSWMYAHBR7VZ5NQNQXSBT37337PZ3FF2OPYFDJWA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.376564159 (556h11m21.247647204s) [seg_pos={0 0}, upload=487612]: Segment reached optimal threshold of 13 pieces. Dec 28 19:40:21.376815168 (556h11m21.247898213s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection Dec 28 19:40:21.376822422 (556h11m21.247905457s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.376948782 (556h11m21.248031827s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection Dec 28 19:40:21.376957529 (556h11m21.248040574s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.377037261 (556h11m21.248120316s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:21.377074962 (556h11m21.248158007s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.377115519 (556h11m21.248198564s) [seg_pos={0 0}, upload=487612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:40:21.377337853 (556h11m21.248420898s) [upload=487612]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:21.377346359 (556h11m21.248429404s) [upload=487612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:21.673294932 (556h11m21.544377977s) [upload=487612]: Done uploading segments... Dec 28 19:40:21.673320830 (556h11m21.544403875s) [upload=487612]: Upload finished. err: <nil> Duration: 11.888385741s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X......X.....X.XX....XXX..X..X......XX.............] duration: 24.539787137s trace: https://jaeger.dev.storj.io/trace/59a4a4c6f8437ff6 FAIL (2025-12-28 19:50:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA) context deadline exceeded ======LOGS====== g segments... Dec 28 19:40:18.799221148 (556h11m18.670304183s) [upload=487612]: Got next segment. Inline: false Dec 28 19:40:18.799268729 (556h11m18.670351765s) [upload=487612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:40:18.799274560 (556h11m18.670357595s) [upload=487612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:40:19.046301604 (556h11m18.917384639s) [seg_pos={0 0}, upload=487612]: Begin upload segment... Dec 28 19:40:19.046366318 (556h11m18.917449363s) [seg_pos={0 0}, upload=487612]: Done begin upload segment. Dec 28 19:40:19.046427104 (556h11m18.917510139s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.046454065 (556h11m18.9175371s) [noise=true , address=145.79.227.6:4257 , piece_id=WAMRDM5M4SS3G75G25AYB2SENETYLZIMK4KNMA33UPXNNZMH4F2Q, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.047153687 (556h11m18.918236732s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.047726838 (556h11m18.918809873s) [noise=true , address=145.79.227.14:4838 , piece_id=XM5TUQ5HOKPBXW4TY4GS57UYQGXMRNYOFQUSOCPQLYRY7TB2J3KQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.048161617 (556h11m18.919244662s) [noise=true , address=145.79.227.15:4874 , piece_id=IQJQRSB5K2ICUNEWM3JYV5G7YKERKLKZAFXQ7ZIXJFU3ZPTZ6VPA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.048604028 (556h11m18.919687073s) [noise=true , address=145.79.227.13:4802 , piece_id=CMYYN6ANGBTFOSYKDUKPLHUBAGQ64EVK2ZCRCQL4GRFFI64QO4OA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.049957866 (556h11m18.921040911s) [noise=true , address=145.79.227.45:4229 , piece_id=N2ZOTUO37BFFTIE5MZQ45K7IJRAPDCEU5OSBYMAFZFPX5DI3C23Q, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.050387555 (556h11m18.9214706s) [noise=true , address=45.91.116.7:4009 , piece_id=GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.051621424 (556h11m18.922704459s) [noise=true , address=77.90.31.11:4555 , piece_id=KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.052026706 (556h11m18.923109741s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.052733111 (556h11m18.923816156s) [noise=true , address=145.79.227.24:4582 , piece_id=C2H5V4G5T3HPD3RL5BGL6IYWVPG4LS6JT5IEVKHXKC5IISBAN7RA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.053928258 (556h11m18.925011303s) [noise=true , address=145.79.227.21:4475 , piece_id=AWYU3CVGPDKUUANUYJZUKKB3GPLG7AOUTWX3X6TOHFSJNO6EOGMA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.055729047 (556h11m18.926812082s) [noise=true , address=45.91.116.10:4081 , piece_id=RKYX4LS7VSTTMSWMYAHBR7VZ5NQNQXSBT37337PZ3FF2OPYFDJWA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.056840893 (556h11m18.927923928s) [noise=true , address=45.91.116.12:4228 , piece_id=J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.058069784 (556h11m18.929152819s) [noise=true , address=145.79.227.37:4566 , piece_id=UT3U3SVVNTDS7REKJ4J4WYXYAZNS35YTOEYPU4V65DQW3AIX2RZA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.059195617 (556h11m18.930278652s) [noise=true , address=145.79.227.28:4727 , piece_id=G7THCZWS4A7PVWA4HI3WZGSFVQO4TZKC6BGNVEU4DQO5DMPCFPUQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:20.770961428 (556h11m20.642044463s) [upload=487612]: Next returned nil segment Dec 28 19:40:20.770980875 (556h11m20.64206392s) [upload=487612]: Waiting for error group managing segments... Dec 28 19:40:21.046457965 (556h11m20.917541s) [noise=true , address=145.79.227.15:4874 , piece_id=IQJQRSB5K2ICUNEWM3JYV5G7YKERKLKZAFXQ7ZIXJFU3ZPTZ6VPA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.059462639 (556h11m20.930545684s) [noise=true , address=145.79.227.24:4582 , piece_id=C2H5V4G5T3HPD3RL5BGL6IYWVPG4LS6JT5IEVKHXKC5IISBAN7RA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.061847620 (556h11m20.932930665s) [noise=true , address=145.79.227.21:4475 , piece_id=AWYU3CVGPDKUUANUYJZUKKB3GPLG7AOUTWX3X6TOHFSJNO6EOGMA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.063315404 (556h11m20.934398439s) [noise=true , address=145.79.227.14:4838 , piece_id=XM5TUQ5HOKPBXW4TY4GS57UYQGXMRNYOFQUSOCPQLYRY7TB2J3KQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.064388047 (556h11m20.935471082s) [noise=true , address=145.79.227.6:4257 , piece_id=WAMRDM5M4SS3G75G25AYB2SENETYLZIMK4KNMA33UPXNNZMH4F2Q, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.072512037 (556h11m20.943595072s) [noise=true , address=145.79.227.45:4229 , piece_id=N2ZOTUO37BFFTIE5MZQ45K7IJRAPDCEU5OSBYMAFZFPX5DI3C23Q, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.089000846 (556h11m20.960083881s) [noise=true , address=145.79.227.37:4566 , piece_id=UT3U3SVVNTDS7REKJ4J4WYXYAZNS35YTOEYPU4V65DQW3AIX2RZA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.156103774 (556h11m21.027186809s) [noise=true , address=145.79.227.13:4802 , piece_id=CMYYN6ANGBTFOSYKDUKPLHUBAGQ64EVK2ZCRCQL4GRFFI64QO4OA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.220520528 (556h11m21.091603563s) [noise=true , address=145.79.227.28:4727 , piece_id=G7THCZWS4A7PVWA4HI3WZGSFVQO4TZKC6BGNVEU4DQO5DMPCFPUQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.243044178 (556h11m21.114127223s) [noise=true , address=77.90.31.11:4555 , piece_id=KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.301399419 (556h11m21.172482464s) [noise=true , address=45.91.116.12:4228 , piece_id=J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.309939952 (556h11m21.181022997s) [noise=true , address=45.91.116.7:4009 , piece_id=GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.376542859 (556h11m21.247625904s) [noise=true , address=45.91.116.10:4081 , piece_id=RKYX4LS7VSTTMSWMYAHBR7VZ5NQNQXSBT37337PZ3FF2OPYFDJWA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.376564159 (556h11m21.247647204s) [seg_pos={0 0}, upload=487612]: Segment reached optimal threshold of 13 pieces. Dec 28 19:40:21.376815168 (556h11m21.247898213s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection Dec 28 19:40:21.376822422 (556h11m21.247905457s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.376948782 (556h11m21.248031827s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection Dec 28 19:40:21.376957529 (556h11m21.248040574s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.377037261 (556h11m21.248120316s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:21.377074962 (556h11m21.248158007s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.377115519 (556h11m21.248198564s) [seg_pos={0 0}, upload=487612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:40:21.377337853 (556h11m21.248420898s) [upload=487612]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:21.377346359 (556h11m21.248429404s) [upload=487612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:21.673294932 (556h11m21.544377977s) [upload=487612]: Done uploading segments... Dec 28 19:40:21.673320830 (556h11m21.544403875s) [upload=487612]: Upload finished. err: <nil> Duration: 24.539832704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..............................X...............] duration: 20.761206375s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:50:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 20.761229169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..............................X..............] duration: 47.718381317s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:49:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 47.718396757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X......X.....X.XX....XXX..X..X......XX............] duration: 11.194472023s trace: https://jaeger.dev.storj.io/trace/59a4a4c6f8437ff6 FAIL (2025-12-28 19:49:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA) context deadline exceeded ======LOGS====== g segments... Dec 28 19:40:18.799221148 (556h11m18.670304183s) [upload=487612]: Got next segment. Inline: false Dec 28 19:40:18.799268729 (556h11m18.670351765s) [upload=487612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:40:18.799274560 (556h11m18.670357595s) [upload=487612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:40:19.046301604 (556h11m18.917384639s) [seg_pos={0 0}, upload=487612]: Begin upload segment... Dec 28 19:40:19.046366318 (556h11m18.917449363s) [seg_pos={0 0}, upload=487612]: Done begin upload segment. Dec 28 19:40:19.046427104 (556h11m18.917510139s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.046454065 (556h11m18.9175371s) [noise=true , address=145.79.227.6:4257 , piece_id=WAMRDM5M4SS3G75G25AYB2SENETYLZIMK4KNMA33UPXNNZMH4F2Q, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.047153687 (556h11m18.918236732s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.047726838 (556h11m18.918809873s) [noise=true , address=145.79.227.14:4838 , piece_id=XM5TUQ5HOKPBXW4TY4GS57UYQGXMRNYOFQUSOCPQLYRY7TB2J3KQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.048161617 (556h11m18.919244662s) [noise=true , address=145.79.227.15:4874 , piece_id=IQJQRSB5K2ICUNEWM3JYV5G7YKERKLKZAFXQ7ZIXJFU3ZPTZ6VPA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.048604028 (556h11m18.919687073s) [noise=true , address=145.79.227.13:4802 , piece_id=CMYYN6ANGBTFOSYKDUKPLHUBAGQ64EVK2ZCRCQL4GRFFI64QO4OA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.049957866 (556h11m18.921040911s) [noise=true , address=145.79.227.45:4229 , piece_id=N2ZOTUO37BFFTIE5MZQ45K7IJRAPDCEU5OSBYMAFZFPX5DI3C23Q, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.050387555 (556h11m18.9214706s) [noise=true , address=45.91.116.7:4009 , piece_id=GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.051621424 (556h11m18.922704459s) [noise=true , address=77.90.31.11:4555 , piece_id=KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.052026706 (556h11m18.923109741s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.052733111 (556h11m18.923816156s) [noise=true , address=145.79.227.24:4582 , piece_id=C2H5V4G5T3HPD3RL5BGL6IYWVPG4LS6JT5IEVKHXKC5IISBAN7RA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.053928258 (556h11m18.925011303s) [noise=true , address=145.79.227.21:4475 , piece_id=AWYU3CVGPDKUUANUYJZUKKB3GPLG7AOUTWX3X6TOHFSJNO6EOGMA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.055729047 (556h11m18.926812082s) [noise=true , address=45.91.116.10:4081 , piece_id=RKYX4LS7VSTTMSWMYAHBR7VZ5NQNQXSBT37337PZ3FF2OPYFDJWA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.056840893 (556h11m18.927923928s) [noise=true , address=45.91.116.12:4228 , piece_id=J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.058069784 (556h11m18.929152819s) [noise=true , address=145.79.227.37:4566 , piece_id=UT3U3SVVNTDS7REKJ4J4WYXYAZNS35YTOEYPU4V65DQW3AIX2RZA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.059195617 (556h11m18.930278652s) [noise=true , address=145.79.227.28:4727 , piece_id=G7THCZWS4A7PVWA4HI3WZGSFVQO4TZKC6BGNVEU4DQO5DMPCFPUQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:20.770961428 (556h11m20.642044463s) [upload=487612]: Next returned nil segment Dec 28 19:40:20.770980875 (556h11m20.64206392s) [upload=487612]: Waiting for error group managing segments... Dec 28 19:40:21.046457965 (556h11m20.917541s) [noise=true , address=145.79.227.15:4874 , piece_id=IQJQRSB5K2ICUNEWM3JYV5G7YKERKLKZAFXQ7ZIXJFU3ZPTZ6VPA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.059462639 (556h11m20.930545684s) [noise=true , address=145.79.227.24:4582 , piece_id=C2H5V4G5T3HPD3RL5BGL6IYWVPG4LS6JT5IEVKHXKC5IISBAN7RA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.061847620 (556h11m20.932930665s) [noise=true , address=145.79.227.21:4475 , piece_id=AWYU3CVGPDKUUANUYJZUKKB3GPLG7AOUTWX3X6TOHFSJNO6EOGMA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.063315404 (556h11m20.934398439s) [noise=true , address=145.79.227.14:4838 , piece_id=XM5TUQ5HOKPBXW4TY4GS57UYQGXMRNYOFQUSOCPQLYRY7TB2J3KQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.064388047 (556h11m20.935471082s) [noise=true , address=145.79.227.6:4257 , piece_id=WAMRDM5M4SS3G75G25AYB2SENETYLZIMK4KNMA33UPXNNZMH4F2Q, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.072512037 (556h11m20.943595072s) [noise=true , address=145.79.227.45:4229 , piece_id=N2ZOTUO37BFFTIE5MZQ45K7IJRAPDCEU5OSBYMAFZFPX5DI3C23Q, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.089000846 (556h11m20.960083881s) [noise=true , address=145.79.227.37:4566 , piece_id=UT3U3SVVNTDS7REKJ4J4WYXYAZNS35YTOEYPU4V65DQW3AIX2RZA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.156103774 (556h11m21.027186809s) [noise=true , address=145.79.227.13:4802 , piece_id=CMYYN6ANGBTFOSYKDUKPLHUBAGQ64EVK2ZCRCQL4GRFFI64QO4OA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.220520528 (556h11m21.091603563s) [noise=true , address=145.79.227.28:4727 , piece_id=G7THCZWS4A7PVWA4HI3WZGSFVQO4TZKC6BGNVEU4DQO5DMPCFPUQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.243044178 (556h11m21.114127223s) [noise=true , address=77.90.31.11:4555 , piece_id=KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.301399419 (556h11m21.172482464s) [noise=true , address=45.91.116.12:4228 , piece_id=J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.309939952 (556h11m21.181022997s) [noise=true , address=45.91.116.7:4009 , piece_id=GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.376542859 (556h11m21.247625904s) [noise=true , address=45.91.116.10:4081 , piece_id=RKYX4LS7VSTTMSWMYAHBR7VZ5NQNQXSBT37337PZ3FF2OPYFDJWA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.376564159 (556h11m21.247647204s) [seg_pos={0 0}, upload=487612]: Segment reached optimal threshold of 13 pieces. Dec 28 19:40:21.376815168 (556h11m21.247898213s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection Dec 28 19:40:21.376822422 (556h11m21.247905457s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.376948782 (556h11m21.248031827s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection Dec 28 19:40:21.376957529 (556h11m21.248040574s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.377037261 (556h11m21.248120316s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:21.377074962 (556h11m21.248158007s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.377115519 (556h11m21.248198564s) [seg_pos={0 0}, upload=487612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:40:21.377337853 (556h11m21.248420898s) [upload=487612]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:21.377346359 (556h11m21.248429404s) [upload=487612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:21.673294932 (556h11m21.544377977s) [upload=487612]: Done uploading segments... Dec 28 19:40:21.673320830 (556h11m21.544403875s) [upload=487612]: Upload finished. err: <nil> Duration: 11.194493715s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..............................X.............] duration: 41.223492709s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:48:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 41.223512176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..........X......X.....X.XX....XXX..X..X......XX...........] duration: 5.99437181s trace: https://jaeger.dev.storj.io/trace/59a4a4c6f8437ff6 FAIL (2025-12-28 19:48:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA) context deadline exceeded ======LOGS====== g segments... Dec 28 19:40:18.799221148 (556h11m18.670304183s) [upload=487612]: Got next segment. Inline: false Dec 28 19:40:18.799268729 (556h11m18.670351765s) [upload=487612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:40:18.799274560 (556h11m18.670357595s) [upload=487612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:40:19.046301604 (556h11m18.917384639s) [seg_pos={0 0}, upload=487612]: Begin upload segment... Dec 28 19:40:19.046366318 (556h11m18.917449363s) [seg_pos={0 0}, upload=487612]: Done begin upload segment. Dec 28 19:40:19.046427104 (556h11m18.917510139s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.046454065 (556h11m18.9175371s) [noise=true , address=145.79.227.6:4257 , piece_id=WAMRDM5M4SS3G75G25AYB2SENETYLZIMK4KNMA33UPXNNZMH4F2Q, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.047153687 (556h11m18.918236732s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.047726838 (556h11m18.918809873s) [noise=true , address=145.79.227.14:4838 , piece_id=XM5TUQ5HOKPBXW4TY4GS57UYQGXMRNYOFQUSOCPQLYRY7TB2J3KQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.048161617 (556h11m18.919244662s) [noise=true , address=145.79.227.15:4874 , piece_id=IQJQRSB5K2ICUNEWM3JYV5G7YKERKLKZAFXQ7ZIXJFU3ZPTZ6VPA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.048604028 (556h11m18.919687073s) [noise=true , address=145.79.227.13:4802 , piece_id=CMYYN6ANGBTFOSYKDUKPLHUBAGQ64EVK2ZCRCQL4GRFFI64QO4OA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.049957866 (556h11m18.921040911s) [noise=true , address=145.79.227.45:4229 , piece_id=N2ZOTUO37BFFTIE5MZQ45K7IJRAPDCEU5OSBYMAFZFPX5DI3C23Q, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.050387555 (556h11m18.9214706s) [noise=true , address=45.91.116.7:4009 , piece_id=GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.051621424 (556h11m18.922704459s) [noise=true , address=77.90.31.11:4555 , piece_id=KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.052026706 (556h11m18.923109741s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.052733111 (556h11m18.923816156s) [noise=true , address=145.79.227.24:4582 , piece_id=C2H5V4G5T3HPD3RL5BGL6IYWVPG4LS6JT5IEVKHXKC5IISBAN7RA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.053928258 (556h11m18.925011303s) [noise=true , address=145.79.227.21:4475 , piece_id=AWYU3CVGPDKUUANUYJZUKKB3GPLG7AOUTWX3X6TOHFSJNO6EOGMA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.055729047 (556h11m18.926812082s) [noise=true , address=45.91.116.10:4081 , piece_id=RKYX4LS7VSTTMSWMYAHBR7VZ5NQNQXSBT37337PZ3FF2OPYFDJWA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.056840893 (556h11m18.927923928s) [noise=true , address=45.91.116.12:4228 , piece_id=J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.058069784 (556h11m18.929152819s) [noise=true , address=145.79.227.37:4566 , piece_id=UT3U3SVVNTDS7REKJ4J4WYXYAZNS35YTOEYPU4V65DQW3AIX2RZA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.059195617 (556h11m18.930278652s) [noise=true , address=145.79.227.28:4727 , piece_id=G7THCZWS4A7PVWA4HI3WZGSFVQO4TZKC6BGNVEU4DQO5DMPCFPUQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:20.770961428 (556h11m20.642044463s) [upload=487612]: Next returned nil segment Dec 28 19:40:20.770980875 (556h11m20.64206392s) [upload=487612]: Waiting for error group managing segments... Dec 28 19:40:21.046457965 (556h11m20.917541s) [noise=true , address=145.79.227.15:4874 , piece_id=IQJQRSB5K2ICUNEWM3JYV5G7YKERKLKZAFXQ7ZIXJFU3ZPTZ6VPA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.059462639 (556h11m20.930545684s) [noise=true , address=145.79.227.24:4582 , piece_id=C2H5V4G5T3HPD3RL5BGL6IYWVPG4LS6JT5IEVKHXKC5IISBAN7RA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.061847620 (556h11m20.932930665s) [noise=true , address=145.79.227.21:4475 , piece_id=AWYU3CVGPDKUUANUYJZUKKB3GPLG7AOUTWX3X6TOHFSJNO6EOGMA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.063315404 (556h11m20.934398439s) [noise=true , address=145.79.227.14:4838 , piece_id=XM5TUQ5HOKPBXW4TY4GS57UYQGXMRNYOFQUSOCPQLYRY7TB2J3KQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.064388047 (556h11m20.935471082s) [noise=true , address=145.79.227.6:4257 , piece_id=WAMRDM5M4SS3G75G25AYB2SENETYLZIMK4KNMA33UPXNNZMH4F2Q, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.072512037 (556h11m20.943595072s) [noise=true , address=145.79.227.45:4229 , piece_id=N2ZOTUO37BFFTIE5MZQ45K7IJRAPDCEU5OSBYMAFZFPX5DI3C23Q, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.089000846 (556h11m20.960083881s) [noise=true , address=145.79.227.37:4566 , piece_id=UT3U3SVVNTDS7REKJ4J4WYXYAZNS35YTOEYPU4V65DQW3AIX2RZA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.156103774 (556h11m21.027186809s) [noise=true , address=145.79.227.13:4802 , piece_id=CMYYN6ANGBTFOSYKDUKPLHUBAGQ64EVK2ZCRCQL4GRFFI64QO4OA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.220520528 (556h11m21.091603563s) [noise=true , address=145.79.227.28:4727 , piece_id=G7THCZWS4A7PVWA4HI3WZGSFVQO4TZKC6BGNVEU4DQO5DMPCFPUQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.243044178 (556h11m21.114127223s) [noise=true , address=77.90.31.11:4555 , piece_id=KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.301399419 (556h11m21.172482464s) [noise=true , address=45.91.116.12:4228 , piece_id=J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.309939952 (556h11m21.181022997s) [noise=true , address=45.91.116.7:4009 , piece_id=GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.376542859 (556h11m21.247625904s) [noise=true , address=45.91.116.10:4081 , piece_id=RKYX4LS7VSTTMSWMYAHBR7VZ5NQNQXSBT37337PZ3FF2OPYFDJWA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.376564159 (556h11m21.247647204s) [seg_pos={0 0}, upload=487612]: Segment reached optimal threshold of 13 pieces. Dec 28 19:40:21.376815168 (556h11m21.247898213s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection Dec 28 19:40:21.376822422 (556h11m21.247905457s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.376948782 (556h11m21.248031827s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection Dec 28 19:40:21.376957529 (556h11m21.248040574s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.377037261 (556h11m21.248120316s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:21.377074962 (556h11m21.248158007s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.377115519 (556h11m21.248198564s) [seg_pos={0 0}, upload=487612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:40:21.377337853 (556h11m21.248420898s) [upload=487612]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:21.377346359 (556h11m21.248429404s) [upload=487612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:21.673294932 (556h11m21.544377977s) [upload=487612]: Done uploading segments... Dec 28 19:40:21.673320830 (556h11m21.544403875s) [upload=487612]: Upload finished. err: <nil> Duration: 5.994402158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X..........X......X.....X.XX....XXX..X..X......XX..........] duration: 12.780188621s trace: https://jaeger.dev.storj.io/trace/59a4a4c6f8437ff6 FAIL (2025-12-28 19:48:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA) context deadline exceeded ======LOGS====== g segments... Dec 28 19:40:18.799221148 (556h11m18.670304183s) [upload=487612]: Got next segment. Inline: false Dec 28 19:40:18.799268729 (556h11m18.670351765s) [upload=487612]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 19:40:18.799274560 (556h11m18.670357595s) [upload=487612]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 19:40:19.046301604 (556h11m18.917384639s) [seg_pos={0 0}, upload=487612]: Begin upload segment... Dec 28 19:40:19.046366318 (556h11m18.917449363s) [seg_pos={0 0}, upload=487612]: Done begin upload segment. Dec 28 19:40:19.046427104 (556h11m18.917510139s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.046454065 (556h11m18.9175371s) [noise=true , address=145.79.227.6:4257 , piece_id=WAMRDM5M4SS3G75G25AYB2SENETYLZIMK4KNMA33UPXNNZMH4F2Q, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.047153687 (556h11m18.918236732s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.047726838 (556h11m18.918809873s) [noise=true , address=145.79.227.14:4838 , piece_id=XM5TUQ5HOKPBXW4TY4GS57UYQGXMRNYOFQUSOCPQLYRY7TB2J3KQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.048161617 (556h11m18.919244662s) [noise=true , address=145.79.227.15:4874 , piece_id=IQJQRSB5K2ICUNEWM3JYV5G7YKERKLKZAFXQ7ZIXJFU3ZPTZ6VPA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.048604028 (556h11m18.919687073s) [noise=true , address=145.79.227.13:4802 , piece_id=CMYYN6ANGBTFOSYKDUKPLHUBAGQ64EVK2ZCRCQL4GRFFI64QO4OA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.049957866 (556h11m18.921040911s) [noise=true , address=145.79.227.45:4229 , piece_id=N2ZOTUO37BFFTIE5MZQ45K7IJRAPDCEU5OSBYMAFZFPX5DI3C23Q, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.050387555 (556h11m18.9214706s) [noise=true , address=45.91.116.7:4009 , piece_id=GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.051621424 (556h11m18.922704459s) [noise=true , address=77.90.31.11:4555 , piece_id=KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.052026706 (556h11m18.923109741s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.052733111 (556h11m18.923816156s) [noise=true , address=145.79.227.24:4582 , piece_id=C2H5V4G5T3HPD3RL5BGL6IYWVPG4LS6JT5IEVKHXKC5IISBAN7RA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.053928258 (556h11m18.925011303s) [noise=true , address=145.79.227.21:4475 , piece_id=AWYU3CVGPDKUUANUYJZUKKB3GPLG7AOUTWX3X6TOHFSJNO6EOGMA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.055729047 (556h11m18.926812082s) [noise=true , address=45.91.116.10:4081 , piece_id=RKYX4LS7VSTTMSWMYAHBR7VZ5NQNQXSBT37337PZ3FF2OPYFDJWA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.056840893 (556h11m18.927923928s) [noise=true , address=45.91.116.12:4228 , piece_id=J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.058069784 (556h11m18.929152819s) [noise=true , address=145.79.227.37:4566 , piece_id=UT3U3SVVNTDS7REKJ4J4WYXYAZNS35YTOEYPU4V65DQW3AIX2RZA, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:19.059195617 (556h11m18.930278652s) [noise=true , address=145.79.227.28:4727 , piece_id=G7THCZWS4A7PVWA4HI3WZGSFVQO4TZKC6BGNVEU4DQO5DMPCFPUQ, seg_pos={0 0}, upload=487612]: Uploading piece... Dec 28 19:40:20.770961428 (556h11m20.642044463s) [upload=487612]: Next returned nil segment Dec 28 19:40:20.770980875 (556h11m20.64206392s) [upload=487612]: Waiting for error group managing segments... Dec 28 19:40:21.046457965 (556h11m20.917541s) [noise=true , address=145.79.227.15:4874 , piece_id=IQJQRSB5K2ICUNEWM3JYV5G7YKERKLKZAFXQ7ZIXJFU3ZPTZ6VPA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.059462639 (556h11m20.930545684s) [noise=true , address=145.79.227.24:4582 , piece_id=C2H5V4G5T3HPD3RL5BGL6IYWVPG4LS6JT5IEVKHXKC5IISBAN7RA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.061847620 (556h11m20.932930665s) [noise=true , address=145.79.227.21:4475 , piece_id=AWYU3CVGPDKUUANUYJZUKKB3GPLG7AOUTWX3X6TOHFSJNO6EOGMA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.063315404 (556h11m20.934398439s) [noise=true , address=145.79.227.14:4838 , piece_id=XM5TUQ5HOKPBXW4TY4GS57UYQGXMRNYOFQUSOCPQLYRY7TB2J3KQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.064388047 (556h11m20.935471082s) [noise=true , address=145.79.227.6:4257 , piece_id=WAMRDM5M4SS3G75G25AYB2SENETYLZIMK4KNMA33UPXNNZMH4F2Q, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.072512037 (556h11m20.943595072s) [noise=true , address=145.79.227.45:4229 , piece_id=N2ZOTUO37BFFTIE5MZQ45K7IJRAPDCEU5OSBYMAFZFPX5DI3C23Q, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.089000846 (556h11m20.960083881s) [noise=true , address=145.79.227.37:4566 , piece_id=UT3U3SVVNTDS7REKJ4J4WYXYAZNS35YTOEYPU4V65DQW3AIX2RZA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.156103774 (556h11m21.027186809s) [noise=true , address=145.79.227.13:4802 , piece_id=CMYYN6ANGBTFOSYKDUKPLHUBAGQ64EVK2ZCRCQL4GRFFI64QO4OA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.220520528 (556h11m21.091603563s) [noise=true , address=145.79.227.28:4727 , piece_id=G7THCZWS4A7PVWA4HI3WZGSFVQO4TZKC6BGNVEU4DQO5DMPCFPUQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.243044178 (556h11m21.114127223s) [noise=true , address=77.90.31.11:4555 , piece_id=KG4FQNO7J4NY2NRJIZVOPEF57LTZC3BAIORYS5UDWYNEU6QPE7EA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.301399419 (556h11m21.172482464s) [noise=true , address=45.91.116.12:4228 , piece_id=J2S4FPVYMI2AQSYJKCWMQYDTNR34CCP4GRPUGBYCWXE4O2V5JSRA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.309939952 (556h11m21.181022997s) [noise=true , address=45.91.116.7:4009 , piece_id=GPMYCW5DMMEJ3EZHU42X5XLUGYA6TTGCAHO73F2M4UHAXZMJZ3SA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.376542859 (556h11m21.247625904s) [noise=true , address=45.91.116.10:4081 , piece_id=RKYX4LS7VSTTMSWMYAHBR7VZ5NQNQXSBT37337PZ3FF2OPYFDJWA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: <nil> Dec 28 19:40:21.376564159 (556h11m21.247647204s) [seg_pos={0 0}, upload=487612]: Segment reached optimal threshold of 13 pieces. Dec 28 19:40:21.376815168 (556h11m21.247898213s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42728->45.91.116.11:4119: use of closed network connection Dec 28 19:40:21.376822422 (556h11m21.247905457s) [noise=true , address=45.91.116.11:4119 , piece_id=NMNQLBJFNU3UNG3Z2JURSW5HZJNKVCBZ34JOV3V6EGDNDIX5I4HA, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.376948782 (556h11m21.248031827s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57374->77.90.31.12:4590: use of closed network connection Dec 28 19:40:21.376957529 (556h11m21.248040574s) [noise=true , address=77.90.31.12:4590 , piece_id=YID3M2XWBM4B6NH3OFEDEQYKFSPXKZVAITIGHAF6ELX56SWI6FBQ, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.377037261 (556h11m21.248120316s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:21.377074962 (556h11m21.248158007s) [noise=true , address=45.91.116.8:4047 , piece_id=DZT66GMZWHKWUBHJAETZDYNKU7QTROTYQJAYWWU7A57K7L2KR5RQ, seg_pos={0 0}, upload=487612]: Long tail cancellation due to optimal threshold Dec 28 19:40:21.377115519 (556h11m21.248198564s) [seg_pos={0 0}, upload=487612]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 19:40:21.377337853 (556h11m21.248420898s) [upload=487612]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:21.377346359 (556h11m21.248429404s) [upload=487612]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:21.673294932 (556h11m21.544377977s) [upload=487612]: Done uploading segments... Dec 28 19:40:21.673320830 (556h11m21.544403875s) [upload=487612]: Upload finished. err: <nil> Duration: 12.780220963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..............................X............] duration: 22.975084026s trace: https://jaeger.dev.storj.io/trace/10e1641b228b2a1f FAIL (2025-12-28 19:48:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 40:05.744568479 (556h11m5.615651514s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745099351 (556h11m5.616182396s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.745777401 (556h11m5.616860436s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746374739 (556h11m5.617457784s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.746934555 (556h11m5.618017601s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.747556861 (556h11m5.618640146s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.748964150 (556h11m5.620047195s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.751119314 (556h11m5.622202359s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.753413222 (556h11m5.624496257s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.755230482 (556h11m5.626313527s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.757598362 (556h11m5.628681407s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.759644679 (556h11m5.630727734s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.761217594 (556h11m5.632300639s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.762138507 (556h11m5.633221552s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.764304321 (556h11m5.635387356s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:05.765421217 (556h11m5.636504252s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Uploading piece... Dec 28 19:40:07.082099826 (556h11m6.953182871s) [upload=487609]: Next returned nil segment Dec 28 19:40:07.082116707 (556h11m6.953199752s) [upload=487609]: Waiting for error group managing segments... Dec 28 19:40:07.415884144 (556h11m7.286967179s) [noise=true , address=64.49.13.154:4932 , piece_id=QB5WM5RZ2P2GC6RIGGM34IANVEI2SNDJ2PO556JXKTDBX5HLD4YA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.615434252 (556h11m7.486517297s) [noise=true , address=145.79.227.23:4550 , piece_id=NWIC722LZUHJSAMPWSKCLPRIDGWPVUEYMS5KRAMX3IIMD2KR4XGQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.647182077 (556h11m7.518265112s) [noise=true , address=145.79.227.28:4744 , piece_id=42LII6EOG3CMBTIL5I4RZQPDYU52TKJN3PGWEQXROBYMNDFYIIVA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.678458624 (556h11m7.549541659s) [noise=true , address=145.79.227.46:4626 , piece_id=OQOTWYLVIU7GJBBLYWQOEEFKR3TNAUOXZS6UAFJWJ32SIOTXNXEA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.793406049 (556h11m7.664489084s) [noise=true , address=77.90.31.19:4860 , piece_id=5FUYVK2Z6LBZOAGFI2NOTAAUIV4NMBI5ERMA5RH4BMONNHABZLDQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.802506498 (556h11m7.673589543s) [noise=true , address=77.90.31.21:4896 , piece_id=ATTC7BZUT4HAAEAZCHFZQE4INV2LLSWR2YCUNVA73LPJCW3YL35A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.853663242 (556h11m7.724746287s) [noise=true , address=77.90.31.10:4522 , piece_id=JOHOMHCNYXF4XQGLN2NJUSWSUO3XFBRRMYD27SLIZAK2RCT3ZC2A, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.880639500 (556h11m7.751722535s) [noise=true , address=77.90.31.11:4582 , piece_id=ZRE24R27TX4E5DTV2POX7EJANQ4CM75D6ZOUS6SKM3QHHQCKRBRA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.904835446 (556h11m7.775918491s) [noise=true , address=77.90.31.18:4816 , piece_id=YT3PSYNQTDQBVBRUDNOXJ2XCN6EFQA54WBSNL6MZMXFXR7XNCHIA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:07.988914260 (556h11m7.859997295s) [noise=true , address=77.90.31.15:4722 , piece_id=AF7F2OA247CJIDFTEU6HEOIYGOINEYPMII53BKJQXI7UZ26MJIUA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.484927449 (556h11m8.356010484s) [noise=true , address=145.79.227.35:4510 , piece_id=P275TU367DTYXBQOGGV5THFUPHJUWGSM5IMJQ753VE4MX4IPOONQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.610317849 (556h11m8.481400894s) [noise=true , address=145.79.227.44:4216 , piece_id=GSBWC2LXUOZYOVD55KNQYA5Z3S2TCYROBB5O2XZIIVFUQMJQMYNA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.611969885 (556h11m8.48305292s) [noise=true , address=145.79.227.34:4916 , piece_id=SPYSFWMJ752FC5JGRE36G54S2FHG4DQFCV6CIZONQHJDXDCN52VQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636098801 (556h11m8.507181836s) [noise=true , address=145.79.227.30:4784 , piece_id=MKCRSFWVFVGOAYCK5M66HO3OT7OMYWLVR2KMDXUGBS2VXXNCUSSQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: <nil> Dec 28 19:40:08.636122136 (556h11m8.507205181s) [seg_pos={0 0}, upload=487609]: Segment reached optimal threshold of 14 pieces. Dec 28 19:40:08.636333929 (556h11m8.507416974s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636342175 (556h11m8.50742521s) [noise=true , address=64.49.13.26:4956 , piece_id=FFHUPOOM7PLEABRHFH6H2SI6WXJBU2XFCLZJUBV4CXUJ6EP4PI2Q, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636585959 (556h11m8.507668994s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636592172 (556h11m8.507675207s) [noise=true , address=145.79.227.43:4163 , piece_id=77BKLLVD25UMTDQZW46CA2N225I7O4HCHCRRPQ7HLJGRHOLGSDHA, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636681511 (556h11m8.507764546s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KZ5a6CQ9tUTrC6EYE7M8yg6p9fK95CN8zQLQmP4tdyz4rBfGh): protocol: expected piece hash; context canceled; EOF Dec 28 19:40:08.636687052 (556h11m8.507770087s) [noise=true , address=145.79.227.19:4083 , piece_id=GL5XL7T2QQABD5364AT4AFV6RZNOWCTVNSMAVIFJU5SUQFHOPZKQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636810327 (556h11m8.507893362s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp): protocol: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51210->145.79.227.17:4992: use of closed network connection Dec 28 19:40:08.636815957 (556h11m8.507899002s) [noise=true , address=145.79.227.17:4992 , piece_id=GUF7AUFCFRRPY5PSYS7DRPFQLGT4ZEL5C7P23US2VI54KYWSUCUQ, seg_pos={0 0}, upload=487609]: Long tail cancellation due to optimal threshold Dec 28 19:40:08.636834313 (556h11m8.507917358s) [seg_pos={0 0}, upload=487609]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 19:40:08.636907953 (556h11m8.507990998s) [upload=487609]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 19:40:08.636913644 (556h11m8.507996689s) [upload=487609]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 19:40:08.924991397 (556h11m8.796074442s) [upload=487609]: Done uploading segments... Dec 28 19:40:08.925014742 (556h11m8.796097777s) [upload=487609]: Upload finished. err: <nil> Duration: 22.975102721s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)