Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X....X........................................X....X.] duration: 35.426810719s trace: https://jaeger.dev.storj.io/trace/374687203cfc05ff FAIL (2026-01-01 03:45:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R8K65zsCspu7W2oAnvtpz7zEsQFkDG1e5BAp6faQYMQrkh4oQ, Piece ID: XNMGIGJO5H4HXFYVKB77P44RDX3YJ2TMGPSRH36U6HN47NSAL3LA) context deadline exceeded; piecestore: (Node ID: 12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, Piece ID: F24TCCKS4ZO25LKMOSIGA3FW4I3H73R37PG443XZCQMBNAXMJRVQ) context deadline exceeded; piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: BRIPHYWYV7CE7DEN4DRHEON33RZP6CG4AY4CNSFWWT6QSU2RB6YQ) context deadline exceeded ======LOGS====== SASRFVAMKJ4XPGJI5BZRA5CFLQJHVA, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.283149000 (636h14m41.154232035s) [noise=true , address=145.79.227.30:4774 , piece_id=WTXNFHLRX336YPVB4UFDX5HDCQWGQX2T5ENLTSNQNSEDBLQZBVNQ, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.283589739 (636h14m41.154672774s) [noise=true , address=77.90.31.18:4830 , piece_id=F24TCCKS4ZO25LKMOSIGA3FW4I3H73R37PG443XZCQMBNAXMJRVQ, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.284010640 (636h14m41.155093675s) [noise=true , address=145.79.227.40:4080 , piece_id=RGLQILGSUZV3IURRKW2MFOEBZRFAPVTLBSTXXK5BHEZ6KT3O43XA, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.284737244 (636h14m41.155820289s) [noise=true , address=145.79.227.32:4848 , piece_id=WOYNEW5FKVFJHYT3HYSA5OJIDYJDUS55ZU6ZMDGQMIWK2YM2QJ4Q, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.285879529 (636h14m41.156962564s) [noise=true , address=77.90.31.10:4535 , piece_id=A5TXRGTQ5GKWZUZMEBFQUTMSXN2BTB2MQJFFPTUBK5XKFMAEIVYQ, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.286218315 (636h14m41.15730138s) [noise=true , address=145.79.227.13:4809 , piece_id=3G3NU3YKNIDS6YWTSSDN3YEIQKEZI467R7SN42EEO2CL5T4O6KKQ, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.287832879 (636h14m41.158915924s) [noise=true , address=145.79.227.19:4097 , piece_id=YMVCIGN7NGV4IIP5E64JRERNQ5HWANFPFXUTR7NOQ74MGC7N6XJA, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.289904222 (636h14m41.160987277s) [noise=true , address=145.79.227.33:4900 , piece_id=XNMGIGJO5H4HXFYVKB77P44RDX3YJ2TMGPSRH36U6HN47NSAL3LA, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.291137972 (636h14m41.162221047s) [noise=true , address=145.79.227.28:4757 , piece_id=FAKOZG5JMFJSX47QW7CXFHAYQFQTZA2RZDX7BOQ5CV5WW4TAZOBA, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.292784497 (636h14m41.163867552s) [noise=true , address=145.79.227.10:4425 , piece_id=ZWZW664BQ52B4JEWMEZZLIPQDXMI4DTLNFFTYLVGTWUFM2F2J5EQ, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.294328196 (636h14m41.165411241s) [noise=true , address=145.79.227.35:4502 , piece_id=TDCGMMHJCMACJHYLBKIAGTKN4PON64IEEGB6H374QRUKS6LHKD2Q, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.295778608 (636h14m41.166861643s) [noise=true , address=145.79.227.43:4171 , piece_id=JYROK7W3AJZAFF5G6WGN4PP5BKMIP3LEOCO6QVHRADH6FQJSVY6Q, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.296267239 (636h14m41.167350274s) [noise=true , address=145.79.227.34:4935 , piece_id=M6SSK4KXRX4F6NPJZWLYWSTKBEJALQHEG7SJDUSMZKFF3IAFFT4A, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:42.740912613 (636h14m42.611995658s) [upload=557501]: Next returned nil segment Jan 1 03:43:42.740936949 (636h14m42.612019984s) [upload=557501]: Waiting for error group managing segments... Jan 1 03:43:43.142749327 (636h14m43.013832372s) [noise=true , address=145.79.227.32:4848 , piece_id=WOYNEW5FKVFJHYT3HYSA5OJIDYJDUS55ZU6ZMDGQMIWK2YM2QJ4Q, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.171158181 (636h14m43.042241216s) [noise=true , address=145.79.227.43:4171 , piece_id=JYROK7W3AJZAFF5G6WGN4PP5BKMIP3LEOCO6QVHRADH6FQJSVY6Q, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.171713578 (636h14m43.042796613s) [noise=true , address=145.79.227.13:4809 , piece_id=3G3NU3YKNIDS6YWTSSDN3YEIQKEZI467R7SN42EEO2CL5T4O6KKQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.327114510 (636h14m43.198197545s) [noise=true , address=77.90.31.14:4689 , piece_id=BRIPHYWYV7CE7DEN4DRHEON33RZP6CG4AY4CNSFWWT6QSU2RB6YQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.406627942 (636h14m43.277710977s) [noise=true , address=77.90.31.18:4830 , piece_id=F24TCCKS4ZO25LKMOSIGA3FW4I3H73R37PG443XZCQMBNAXMJRVQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.473717668 (636h14m43.344800704s) [noise=true , address=77.90.31.10:4535 , piece_id=A5TXRGTQ5GKWZUZMEBFQUTMSXN2BTB2MQJFFPTUBK5XKFMAEIVYQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.547083546 (636h14m43.418166591s) [noise=true , address=45.91.116.4:4948 , piece_id=FQRYJFEKJRR5OD4JGYVVLY6CLPHPIQXV3FS7TOE2GYITZS2MIALQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.610473989 (636h14m43.481557034s) [noise=true , address=145.79.227.10:4425 , piece_id=ZWZW664BQ52B4JEWMEZZLIPQDXMI4DTLNFFTYLVGTWUFM2F2J5EQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.055488508 (636h14m43.926571553s) [noise=true , address=145.79.227.49:4350 , piece_id=7RDY7ORRKOOQ3K6WYBJDSWSMAAMCULY5VQ5TMH47MAIPPJ2ZVUSA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.083262794 (636h14m43.95434584s) [noise=true , address=145.79.227.30:4774 , piece_id=WTXNFHLRX336YPVB4UFDX5HDCQWGQX2T5ENLTSNQNSEDBLQZBVNQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.124018758 (636h14m43.995101793s) [noise=true , address=145.79.227.35:4502 , piece_id=TDCGMMHJCMACJHYLBKIAGTKN4PON64IEEGB6H374QRUKS6LHKD2Q, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.130763421 (636h14m44.001846456s) [noise=true , address=145.79.227.34:4935 , piece_id=M6SSK4KXRX4F6NPJZWLYWSTKBEJALQHEG7SJDUSMZKFF3IAFFT4A, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.215973982 (636h14m44.087057027s) [noise=true , address=145.79.227.19:4097 , piece_id=YMVCIGN7NGV4IIP5E64JRERNQ5HWANFPFXUTR7NOQ74MGC7N6XJA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.269896186 (636h14m44.140979221s) [noise=true , address=145.79.227.33:4900 , piece_id=XNMGIGJO5H4HXFYVKB77P44RDX3YJ2TMGPSRH36U6HN47NSAL3LA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.269920352 (636h14m44.141003387s) [seg_pos={0 0}, upload=557501]: Segment reached optimal threshold of 14 pieces. Jan 1 03:43:44.270138146 (636h14m44.141221191s) [noise=true , address=145.79.227.40:4080 , piece_id=RGLQILGSUZV3IURRKW2MFOEBZRFAPVTLBSTXXK5BHEZ6KT3O43XA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C): protocol: expected piece hash; context canceled; EOF Jan 1 03:43:44.270148567 (636h14m44.141231602s) [noise=true , address=145.79.227.40:4080 , piece_id=RGLQILGSUZV3IURRKW2MFOEBZRFAPVTLBSTXXK5BHEZ6KT3O43XA, seg_pos={0 0}, upload=557501]: Long tail cancellation due to optimal threshold Jan 1 03:43:44.270176370 (636h14m44.141259405s) [noise=true , address=145.79.227.9:4391 , piece_id=ZACIRH5LGQEFVPF722QNPXSASRFVAMKJ4XPGJI5BZRA5CFLQJHVA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MHmmjheT4VDX93MiQNzQyTWkxZFzz2PwNvdRfsevZRbj6qmJS): protocol: write tcp 188.166.225.170:34468->145.79.227.9:4391: use of closed network connection; write tcp 188.166.225.170:34468->145.79.227.9:4391: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34468->145.79.227.9:4391: use of closed network connection Jan 1 03:43:44.270185427 (636h14m44.141268462s) [noise=true , address=145.79.227.9:4391 , piece_id=ZACIRH5LGQEFVPF722QNPXSASRFVAMKJ4XPGJI5BZRA5CFLQJHVA, seg_pos={0 0}, upload=557501]: Long tail cancellation due to optimal threshold Jan 1 03:43:44.270275979 (636h14m44.141359014s) [noise=true , address=145.79.227.28:4757 , piece_id=FAKOZG5JMFJSX47QW7CXFHAYQFQTZA2RZDX7BOQ5CV5WW4TAZOBA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18M8G3TUJ9jLNDnWMpYaQBXfBUXpD7hzf4Xt6BzasYNnsfTWxS): protocol: write tcp 188.166.225.170:52798->145.79.227.28:4757: use of closed network connection; write tcp 188.166.225.170:52798->145.79.227.28:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52798->145.79.227.28:4757: use of closed network connection Jan 1 03:43:44.270284015 (636h14m44.14136706s) [noise=true , address=145.79.227.28:4757 , piece_id=FAKOZG5JMFJSX47QW7CXFHAYQFQTZA2RZDX7BOQ5CV5WW4TAZOBA, seg_pos={0 0}, upload=557501]: Long tail cancellation due to optimal threshold Jan 1 03:43:44.270292641 (636h14m44.141375676s) [noise=true , address=64.49.13.169:4716 , piece_id=QLDH7SBU7PC52KO4G6XOE2CXNCFFNAG7JGKYECPBH4SRCQFXOLKQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:39492->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:39492->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39492->64.49.13.169:4716: use of closed network connection Jan 1 03:43:44.270299985 (636h14m44.14138302s) [noise=true , address=64.49.13.169:4716 , piece_id=QLDH7SBU7PC52KO4G6XOE2CXNCFFNAG7JGKYECPBH4SRCQFXOLKQ, seg_pos={0 0}, upload=557501]: Long tail cancellation due to optimal threshold Jan 1 03:43:44.270315384 (636h14m44.14139842s) [seg_pos={0 0}, upload=557501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:43:44.270402259 (636h14m44.141485294s) [upload=557501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:43:44.270419662 (636h14m44.141502697s) [upload=557501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:43:44.564652596 (636h14m44.435735641s) [upload=557501]: Done uploading segments... Jan 1 03:43:44.564680559 (636h14m44.435763605s) [upload=557501]: Upload finished. err: <nil> Duration: 35.426826519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X......X............................X........X..X...X.] duration: 3.675799494s trace: https://jaeger.dev.storj.io/trace/43ebb72ee5f505b8 FAIL (2026-01-01 03:45:09 UTC) orbiter warning: context deadline exceeded Duration: 3.675827447s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX...X.X..] duration: 6.011613624s trace: https://jaeger.dev.storj.io/trace/1c54f72d9490089d FAIL (2026-01-01 03:44:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: FBLNPCSXSNK4JGPJ326ER7T6ZOFND33MYWQRWESSTTRI7HILXQOA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 7CO63CDAFPC6EKDNMGXMTSCUFJFX7H4XFLKFSL3M25D3ET4OUDWQ) context deadline exceeded ======LOGS====== Jan 1 03:42:55.370396032 (636h13m55.241479077s) [upload=557491]: Starting upload Jan 1 03:42:55.370481203 (636h13m55.241564248s) [upload=557491]: Done starting upload Jan 1 03:42:55.370558051 (636h13m55.241641086s) [upload=557491]: Uploading segments... Jan 1 03:42:55.370727773 (636h13m55.241810818s) [upload=557491]: Got next segment. Inline: false Jan 1 03:42:55.370768892 (636h13m55.241851927s) [upload=557491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:42:55.370774262 (636h13m55.241857297s) [upload=557491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:42:55.615244972 (636h13m55.486328037s) [seg_pos={0 0}, upload=557491]: Begin upload segment... Jan 1 03:42:55.615319634 (636h13m55.486402669s) [seg_pos={0 0}, upload=557491]: Done begin upload segment. Jan 1 03:42:55.615442186 (636h13m55.486525222s) [noise=true , address=145.79.227.19:4077 , piece_id=3QYQIMFOMTFCXVRV2OPQZHYCV7SYXLPF3B2U4H56NGYJ7LCQDHXQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.615495849 (636h13m55.486578904s) [noise=true , address=145.79.227.36:4519 , piece_id=H5SGXIA3YRSEKQURYPTVRJ5FY2BDSSBYFVJNYUSL2VDRGBOWQPEQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.616143773 (636h13m55.487226818s) [noise=true , address=145.79.227.17:4970 , piece_id=KM4GYXML3JJ3XKA4WW4UQ67XQ7VUR64YWMFDHZB2YZXJRNNVKF3Q, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.616856400 (636h13m55.487939435s) [noise=true , address=145.79.227.4:4187 , piece_id=QKE5QSIHAFU7QHQAGXBRWHDPQB6OID5X46NQKLCKE666XBRURUYA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.617558527 (636h13m55.488641562s) [noise=true , address=77.90.31.21:4878 , piece_id=FBLNPCSXSNK4JGPJ326ER7T6ZOFND33MYWQRWESSTTRI7HILXQOA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.618241337 (636h13m55.489324382s) [noise=true , address=145.79.227.15:4875 , piece_id=Y5FT7KRMZRSCLVOSQE3MJPIMQK6CJMGNQNNSWFHINYPGD2WAACHA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.618847992 (636h13m55.489931037s) [noise=true , address=45.91.116.12:4226 , piece_id=DFMRPG3SR4XVT6DLMHUSKZ7ZKOJZ2NV7MAEPVQ5QRPKFSXV3ZLHA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.619355048 (636h13m55.490438093s) [noise=true , address=145.79.227.51:4411 , piece_id=XDHEPJUSHDRYEAABUXO7P3SX6CTJCN3WFE6CEVS4MBREC3PZGZMQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.620738171 (636h13m55.491821246s) [noise=true , address=145.79.227.39:4013 , piece_id=CW4ZOVRJLSB2ORCD3EC7TPFGSTMM3GTMQI45QOM4VTKTAV6OKVYA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.620906642 (636h13m55.491989677s) [noise=true , address=77.90.31.19:4842 , piece_id=HNKE2G6U56TH7IRXOEQVDDK2AGJFP2JXMM6PRLEL6ZFUP757EDVQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.622842278 (636h13m55.493925313s) [noise=true , address=45.91.116.10:4081 , piece_id=7CO63CDAFPC6EKDNMGXMTSCUFJFX7H4XFLKFSL3M25D3ET4OUDWQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.623066144 (636h13m55.494149179s) [noise=true , address=145.79.227.20:4439 , piece_id=GW5FHVJDWNUB72SEFIZ4L77K4OWY5T7IAHE3HCCWRLS2BSRAX2KQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.624645010 (636h13m55.495728055s) [noise=true , address=77.90.31.10:4519 , piece_id=YZBM7ZVYOHTBDUFMFPVOYDWSYII5IYJOVNFZAT5VJ24OWAQLJYIA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.634717239 (636h13m55.505800284s) [noise=true , address=145.79.227.14:4839 , piece_id=HNZBYCAFHIBJEU2PV2MUZ2FGE3O2BHTIX4LXMSUAVSKKICCPKDZQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.638540770 (636h13m55.509623815s) [noise=true , address=77.90.31.12:4589 , piece_id=7ERRHJAUHFPWAI5CSBHSMIBV62BNGGTAXIBETXCYGDNP23VLDKCQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.640554774 (636h13m55.511637819s) [noise=true , address=45.91.116.8:4046 , piece_id=VGW2MI5LCOY5M724R6FKXXYWX3ZUZWGRJAP4OAUCZEGGKGE6JWZQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:57.330501460 (636h13m57.201584496s) [upload=557491]: Next returned nil segment Jan 1 03:42:57.330514725 (636h13m57.20159776s) [upload=557491]: Waiting for error group managing segments... Jan 1 03:42:57.609560712 (636h13m57.480643747s) [noise=true , address=145.79.227.36:4519 , piece_id=H5SGXIA3YRSEKQURYPTVRJ5FY2BDSSBYFVJNYUSL2VDRGBOWQPEQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.620460857 (636h13m57.491543892s) [noise=true , address=145.79.227.20:4439 , piece_id=GW5FHVJDWNUB72SEFIZ4L77K4OWY5T7IAHE3HCCWRLS2BSRAX2KQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.621281650 (636h13m57.492364685s) [noise=true , address=145.79.227.15:4875 , piece_id=Y5FT7KRMZRSCLVOSQE3MJPIMQK6CJMGNQNNSWFHINYPGD2WAACHA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.626638129 (636h13m57.497721174s) [noise=true , address=145.79.227.51:4411 , piece_id=XDHEPJUSHDRYEAABUXO7P3SX6CTJCN3WFE6CEVS4MBREC3PZGZMQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.634713907 (636h13m57.505796952s) [noise=true , address=145.79.227.4:4187 , piece_id=QKE5QSIHAFU7QHQAGXBRWHDPQB6OID5X46NQKLCKE666XBRURUYA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.634782467 (636h13m57.505865512s) [noise=true , address=145.79.227.17:4970 , piece_id=KM4GYXML3JJ3XKA4WW4UQ67XQ7VUR64YWMFDHZB2YZXJRNNVKF3Q, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.635410192 (636h13m57.506493237s) [noise=true , address=145.79.227.19:4077 , piece_id=3QYQIMFOMTFCXVRV2OPQZHYCV7SYXLPF3B2U4H56NGYJ7LCQDHXQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.646671215 (636h13m57.51775426s) [noise=true , address=145.79.227.39:4013 , piece_id=CW4ZOVRJLSB2ORCD3EC7TPFGSTMM3GTMQI45QOM4VTKTAV6OKVYA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.724274681 (636h13m57.595357716s) [noise=true , address=145.79.227.14:4839 , piece_id=HNZBYCAFHIBJEU2PV2MUZ2FGE3O2BHTIX4LXMSUAVSKKICCPKDZQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.877926331 (636h13m57.749009376s) [noise=true , address=77.90.31.21:4878 , piece_id=FBLNPCSXSNK4JGPJ326ER7T6ZOFND33MYWQRWESSTTRI7HILXQOA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.923048219 (636h13m57.794131304s) [noise=true , address=77.90.31.19:4842 , piece_id=HNKE2G6U56TH7IRXOEQVDDK2AGJFP2JXMM6PRLEL6ZFUP757EDVQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.936788983 (636h13m57.807872018s) [noise=true , address=45.91.116.12:4226 , piece_id=DFMRPG3SR4XVT6DLMHUSKZ7ZKOJZ2NV7MAEPVQ5QRPKFSXV3ZLHA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.947857278 (636h13m57.818940313s) [noise=true , address=45.91.116.10:4081 , piece_id=7CO63CDAFPC6EKDNMGXMTSCUFJFX7H4XFLKFSL3M25D3ET4OUDWQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.947876315 (636h13m57.81895936s) [seg_pos={0 0}, upload=557491]: Segment reached optimal threshold of 13 pieces. Jan 1 03:42:57.948102415 (636h13m57.81918545s) [noise=true , address=77.90.31.10:4519 , piece_id=YZBM7ZVYOHTBDUFMFPVOYDWSYII5IYJOVNFZAT5VJ24OWAQLJYIA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 1 03:42:57.948110320 (636h13m57.819193355s) [noise=true , address=77.90.31.10:4519 , piece_id=YZBM7ZVYOHTBDUFMFPVOYDWSYII5IYJOVNFZAT5VJ24OWAQLJYIA, seg_pos={0 0}, upload=557491]: Long tail cancellation due to optimal threshold Jan 1 03:42:57.948337112 (636h13m57.819420147s) [noise=true , address=45.91.116.8:4046 , piece_id=VGW2MI5LCOY5M724R6FKXXYWX3ZUZWGRJAP4OAUCZEGGKGE6JWZQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Jan 1 03:42:57.948343814 (636h13m57.81942685s) [noise=true , address=45.91.116.8:4046 , piece_id=VGW2MI5LCOY5M724R6FKXXYWX3ZUZWGRJAP4OAUCZEGGKGE6JWZQ, seg_pos={0 0}, upload=557491]: Long tail cancellation due to optimal threshold Jan 1 03:42:57.948452912 (636h13m57.819535957s) [noise=true , address=77.90.31.12:4589 , piece_id=7ERRHJAUHFPWAI5CSBHSMIBV62BNGGTAXIBETXCYGDNP23VLDKCQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jan 1 03:42:57.948496194 (636h13m57.819579229s) [noise=true , address=77.90.31.12:4589 , piece_id=7ERRHJAUHFPWAI5CSBHSMIBV62BNGGTAXIBETXCYGDNP23VLDKCQ, seg_pos={0 0}, upload=557491]: Long tail cancellation due to optimal threshold Jan 1 03:42:57.948530439 (636h13m57.819613475s) [seg_pos={0 0}, upload=557491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:42:57.948649797 (636h13m57.819732842s) [upload=557491]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:42:57.948657161 (636h13m57.819740196s) [upload=557491]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:42:58.242849077 (636h13m58.113932132s) [upload=557491]: Done uploading segments... Jan 1 03:42:58.242870609 (636h13m58.113953644s) [upload=557491]: Upload finished. err: <nil> Duration: 6.01165334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X....X........................................X....X] duration: 1m0.004265077s trace: https://jaeger.dev.storj.io/trace/374687203cfc05ff FAIL (2026-01-01 03:44:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1R8K65zsCspu7W2oAnvtpz7zEsQFkDG1e5BAp6faQYMQrkh4oQ, Piece ID: XNMGIGJO5H4HXFYVKB77P44RDX3YJ2TMGPSRH36U6HN47NSAL3LA) context deadline exceeded; piecestore: (Node ID: 12U2TBUpmxK1xeXXWHv1VNE8xgjQGU3MrcKsEmDRtPvpj2Tnddz, Piece ID: F24TCCKS4ZO25LKMOSIGA3FW4I3H73R37PG443XZCQMBNAXMJRVQ) context deadline exceeded; piecestore: (Node ID: 1bzvSPximqxjmXA4xtXZzD8Xyp27KURJ3oH1zCvbPmPwaLkuhM, Piece ID: BRIPHYWYV7CE7DEN4DRHEON33RZP6CG4AY4CNSFWWT6QSU2RB6YQ) context deadline exceeded ======LOGS====== SASRFVAMKJ4XPGJI5BZRA5CFLQJHVA, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.283149000 (636h14m41.154232035s) [noise=true , address=145.79.227.30:4774 , piece_id=WTXNFHLRX336YPVB4UFDX5HDCQWGQX2T5ENLTSNQNSEDBLQZBVNQ, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.283589739 (636h14m41.154672774s) [noise=true , address=77.90.31.18:4830 , piece_id=F24TCCKS4ZO25LKMOSIGA3FW4I3H73R37PG443XZCQMBNAXMJRVQ, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.284010640 (636h14m41.155093675s) [noise=true , address=145.79.227.40:4080 , piece_id=RGLQILGSUZV3IURRKW2MFOEBZRFAPVTLBSTXXK5BHEZ6KT3O43XA, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.284737244 (636h14m41.155820289s) [noise=true , address=145.79.227.32:4848 , piece_id=WOYNEW5FKVFJHYT3HYSA5OJIDYJDUS55ZU6ZMDGQMIWK2YM2QJ4Q, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.285879529 (636h14m41.156962564s) [noise=true , address=77.90.31.10:4535 , piece_id=A5TXRGTQ5GKWZUZMEBFQUTMSXN2BTB2MQJFFPTUBK5XKFMAEIVYQ, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.286218315 (636h14m41.15730138s) [noise=true , address=145.79.227.13:4809 , piece_id=3G3NU3YKNIDS6YWTSSDN3YEIQKEZI467R7SN42EEO2CL5T4O6KKQ, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.287832879 (636h14m41.158915924s) [noise=true , address=145.79.227.19:4097 , piece_id=YMVCIGN7NGV4IIP5E64JRERNQ5HWANFPFXUTR7NOQ74MGC7N6XJA, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.289904222 (636h14m41.160987277s) [noise=true , address=145.79.227.33:4900 , piece_id=XNMGIGJO5H4HXFYVKB77P44RDX3YJ2TMGPSRH36U6HN47NSAL3LA, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.291137972 (636h14m41.162221047s) [noise=true , address=145.79.227.28:4757 , piece_id=FAKOZG5JMFJSX47QW7CXFHAYQFQTZA2RZDX7BOQ5CV5WW4TAZOBA, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.292784497 (636h14m41.163867552s) [noise=true , address=145.79.227.10:4425 , piece_id=ZWZW664BQ52B4JEWMEZZLIPQDXMI4DTLNFFTYLVGTWUFM2F2J5EQ, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.294328196 (636h14m41.165411241s) [noise=true , address=145.79.227.35:4502 , piece_id=TDCGMMHJCMACJHYLBKIAGTKN4PON64IEEGB6H374QRUKS6LHKD2Q, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.295778608 (636h14m41.166861643s) [noise=true , address=145.79.227.43:4171 , piece_id=JYROK7W3AJZAFF5G6WGN4PP5BKMIP3LEOCO6QVHRADH6FQJSVY6Q, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:41.296267239 (636h14m41.167350274s) [noise=true , address=145.79.227.34:4935 , piece_id=M6SSK4KXRX4F6NPJZWLYWSTKBEJALQHEG7SJDUSMZKFF3IAFFT4A, seg_pos={0 0}, upload=557501]: Uploading piece... Jan 1 03:43:42.740912613 (636h14m42.611995658s) [upload=557501]: Next returned nil segment Jan 1 03:43:42.740936949 (636h14m42.612019984s) [upload=557501]: Waiting for error group managing segments... Jan 1 03:43:43.142749327 (636h14m43.013832372s) [noise=true , address=145.79.227.32:4848 , piece_id=WOYNEW5FKVFJHYT3HYSA5OJIDYJDUS55ZU6ZMDGQMIWK2YM2QJ4Q, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.171158181 (636h14m43.042241216s) [noise=true , address=145.79.227.43:4171 , piece_id=JYROK7W3AJZAFF5G6WGN4PP5BKMIP3LEOCO6QVHRADH6FQJSVY6Q, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.171713578 (636h14m43.042796613s) [noise=true , address=145.79.227.13:4809 , piece_id=3G3NU3YKNIDS6YWTSSDN3YEIQKEZI467R7SN42EEO2CL5T4O6KKQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.327114510 (636h14m43.198197545s) [noise=true , address=77.90.31.14:4689 , piece_id=BRIPHYWYV7CE7DEN4DRHEON33RZP6CG4AY4CNSFWWT6QSU2RB6YQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.406627942 (636h14m43.277710977s) [noise=true , address=77.90.31.18:4830 , piece_id=F24TCCKS4ZO25LKMOSIGA3FW4I3H73R37PG443XZCQMBNAXMJRVQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.473717668 (636h14m43.344800704s) [noise=true , address=77.90.31.10:4535 , piece_id=A5TXRGTQ5GKWZUZMEBFQUTMSXN2BTB2MQJFFPTUBK5XKFMAEIVYQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.547083546 (636h14m43.418166591s) [noise=true , address=45.91.116.4:4948 , piece_id=FQRYJFEKJRR5OD4JGYVVLY6CLPHPIQXV3FS7TOE2GYITZS2MIALQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:43.610473989 (636h14m43.481557034s) [noise=true , address=145.79.227.10:4425 , piece_id=ZWZW664BQ52B4JEWMEZZLIPQDXMI4DTLNFFTYLVGTWUFM2F2J5EQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.055488508 (636h14m43.926571553s) [noise=true , address=145.79.227.49:4350 , piece_id=7RDY7ORRKOOQ3K6WYBJDSWSMAAMCULY5VQ5TMH47MAIPPJ2ZVUSA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.083262794 (636h14m43.95434584s) [noise=true , address=145.79.227.30:4774 , piece_id=WTXNFHLRX336YPVB4UFDX5HDCQWGQX2T5ENLTSNQNSEDBLQZBVNQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.124018758 (636h14m43.995101793s) [noise=true , address=145.79.227.35:4502 , piece_id=TDCGMMHJCMACJHYLBKIAGTKN4PON64IEEGB6H374QRUKS6LHKD2Q, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.130763421 (636h14m44.001846456s) [noise=true , address=145.79.227.34:4935 , piece_id=M6SSK4KXRX4F6NPJZWLYWSTKBEJALQHEG7SJDUSMZKFF3IAFFT4A, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.215973982 (636h14m44.087057027s) [noise=true , address=145.79.227.19:4097 , piece_id=YMVCIGN7NGV4IIP5E64JRERNQ5HWANFPFXUTR7NOQ74MGC7N6XJA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.269896186 (636h14m44.140979221s) [noise=true , address=145.79.227.33:4900 , piece_id=XNMGIGJO5H4HXFYVKB77P44RDX3YJ2TMGPSRH36U6HN47NSAL3LA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: <nil> Jan 1 03:43:44.269920352 (636h14m44.141003387s) [seg_pos={0 0}, upload=557501]: Segment reached optimal threshold of 14 pieces. Jan 1 03:43:44.270138146 (636h14m44.141221191s) [noise=true , address=145.79.227.40:4080 , piece_id=RGLQILGSUZV3IURRKW2MFOEBZRFAPVTLBSTXXK5BHEZ6KT3O43XA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C): protocol: expected piece hash; context canceled; EOF Jan 1 03:43:44.270148567 (636h14m44.141231602s) [noise=true , address=145.79.227.40:4080 , piece_id=RGLQILGSUZV3IURRKW2MFOEBZRFAPVTLBSTXXK5BHEZ6KT3O43XA, seg_pos={0 0}, upload=557501]: Long tail cancellation due to optimal threshold Jan 1 03:43:44.270176370 (636h14m44.141259405s) [noise=true , address=145.79.227.9:4391 , piece_id=ZACIRH5LGQEFVPF722QNPXSASRFVAMKJ4XPGJI5BZRA5CFLQJHVA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12MHmmjheT4VDX93MiQNzQyTWkxZFzz2PwNvdRfsevZRbj6qmJS): protocol: write tcp 188.166.225.170:34468->145.79.227.9:4391: use of closed network connection; write tcp 188.166.225.170:34468->145.79.227.9:4391: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34468->145.79.227.9:4391: use of closed network connection Jan 1 03:43:44.270185427 (636h14m44.141268462s) [noise=true , address=145.79.227.9:4391 , piece_id=ZACIRH5LGQEFVPF722QNPXSASRFVAMKJ4XPGJI5BZRA5CFLQJHVA, seg_pos={0 0}, upload=557501]: Long tail cancellation due to optimal threshold Jan 1 03:43:44.270275979 (636h14m44.141359014s) [noise=true , address=145.79.227.28:4757 , piece_id=FAKOZG5JMFJSX47QW7CXFHAYQFQTZA2RZDX7BOQ5CV5WW4TAZOBA, seg_pos={0 0}, upload=557501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18M8G3TUJ9jLNDnWMpYaQBXfBUXpD7hzf4Xt6BzasYNnsfTWxS): protocol: write tcp 188.166.225.170:52798->145.79.227.28:4757: use of closed network connection; write tcp 188.166.225.170:52798->145.79.227.28:4757: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52798->145.79.227.28:4757: use of closed network connection Jan 1 03:43:44.270284015 (636h14m44.14136706s) [noise=true , address=145.79.227.28:4757 , piece_id=FAKOZG5JMFJSX47QW7CXFHAYQFQTZA2RZDX7BOQ5CV5WW4TAZOBA, seg_pos={0 0}, upload=557501]: Long tail cancellation due to optimal threshold Jan 1 03:43:44.270292641 (636h14m44.141375676s) [noise=true , address=64.49.13.169:4716 , piece_id=QLDH7SBU7PC52KO4G6XOE2CXNCFFNAG7JGKYECPBH4SRCQFXOLKQ, seg_pos={0 0}, upload=557501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:39492->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:39492->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39492->64.49.13.169:4716: use of closed network connection Jan 1 03:43:44.270299985 (636h14m44.14138302s) [noise=true , address=64.49.13.169:4716 , piece_id=QLDH7SBU7PC52KO4G6XOE2CXNCFFNAG7JGKYECPBH4SRCQFXOLKQ, seg_pos={0 0}, upload=557501]: Long tail cancellation due to optimal threshold Jan 1 03:43:44.270315384 (636h14m44.14139842s) [seg_pos={0 0}, upload=557501]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:43:44.270402259 (636h14m44.141485294s) [upload=557501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:43:44.270419662 (636h14m44.141502697s) [upload=557501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:43:44.564652596 (636h14m44.435735641s) [upload=557501]: Done uploading segments... Jan 1 03:43:44.564680559 (636h14m44.435763605s) [upload=557501]: Upload finished. err: <nil> Duration: 1m0.00427659s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X......X............................X........X..X...X] duration: 5.000868891s trace: https://jaeger.dev.storj.io/trace/43ebb72ee5f505b8 FAIL (2026-01-01 03:44:16 UTC) orbiter warning: context deadline exceeded Duration: 5.000897996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX...X.X.] duration: 6.097082097s trace: https://jaeger.dev.storj.io/trace/1c54f72d9490089d FAIL (2026-01-01 03:44:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: FBLNPCSXSNK4JGPJ326ER7T6ZOFND33MYWQRWESSTTRI7HILXQOA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 7CO63CDAFPC6EKDNMGXMTSCUFJFX7H4XFLKFSL3M25D3ET4OUDWQ) context deadline exceeded ======LOGS====== Jan 1 03:42:55.370396032 (636h13m55.241479077s) [upload=557491]: Starting upload Jan 1 03:42:55.370481203 (636h13m55.241564248s) [upload=557491]: Done starting upload Jan 1 03:42:55.370558051 (636h13m55.241641086s) [upload=557491]: Uploading segments... Jan 1 03:42:55.370727773 (636h13m55.241810818s) [upload=557491]: Got next segment. Inline: false Jan 1 03:42:55.370768892 (636h13m55.241851927s) [upload=557491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:42:55.370774262 (636h13m55.241857297s) [upload=557491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:42:55.615244972 (636h13m55.486328037s) [seg_pos={0 0}, upload=557491]: Begin upload segment... Jan 1 03:42:55.615319634 (636h13m55.486402669s) [seg_pos={0 0}, upload=557491]: Done begin upload segment. Jan 1 03:42:55.615442186 (636h13m55.486525222s) [noise=true , address=145.79.227.19:4077 , piece_id=3QYQIMFOMTFCXVRV2OPQZHYCV7SYXLPF3B2U4H56NGYJ7LCQDHXQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.615495849 (636h13m55.486578904s) [noise=true , address=145.79.227.36:4519 , piece_id=H5SGXIA3YRSEKQURYPTVRJ5FY2BDSSBYFVJNYUSL2VDRGBOWQPEQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.616143773 (636h13m55.487226818s) [noise=true , address=145.79.227.17:4970 , piece_id=KM4GYXML3JJ3XKA4WW4UQ67XQ7VUR64YWMFDHZB2YZXJRNNVKF3Q, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.616856400 (636h13m55.487939435s) [noise=true , address=145.79.227.4:4187 , piece_id=QKE5QSIHAFU7QHQAGXBRWHDPQB6OID5X46NQKLCKE666XBRURUYA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.617558527 (636h13m55.488641562s) [noise=true , address=77.90.31.21:4878 , piece_id=FBLNPCSXSNK4JGPJ326ER7T6ZOFND33MYWQRWESSTTRI7HILXQOA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.618241337 (636h13m55.489324382s) [noise=true , address=145.79.227.15:4875 , piece_id=Y5FT7KRMZRSCLVOSQE3MJPIMQK6CJMGNQNNSWFHINYPGD2WAACHA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.618847992 (636h13m55.489931037s) [noise=true , address=45.91.116.12:4226 , piece_id=DFMRPG3SR4XVT6DLMHUSKZ7ZKOJZ2NV7MAEPVQ5QRPKFSXV3ZLHA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.619355048 (636h13m55.490438093s) [noise=true , address=145.79.227.51:4411 , piece_id=XDHEPJUSHDRYEAABUXO7P3SX6CTJCN3WFE6CEVS4MBREC3PZGZMQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.620738171 (636h13m55.491821246s) [noise=true , address=145.79.227.39:4013 , piece_id=CW4ZOVRJLSB2ORCD3EC7TPFGSTMM3GTMQI45QOM4VTKTAV6OKVYA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.620906642 (636h13m55.491989677s) [noise=true , address=77.90.31.19:4842 , piece_id=HNKE2G6U56TH7IRXOEQVDDK2AGJFP2JXMM6PRLEL6ZFUP757EDVQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.622842278 (636h13m55.493925313s) [noise=true , address=45.91.116.10:4081 , piece_id=7CO63CDAFPC6EKDNMGXMTSCUFJFX7H4XFLKFSL3M25D3ET4OUDWQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.623066144 (636h13m55.494149179s) [noise=true , address=145.79.227.20:4439 , piece_id=GW5FHVJDWNUB72SEFIZ4L77K4OWY5T7IAHE3HCCWRLS2BSRAX2KQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.624645010 (636h13m55.495728055s) [noise=true , address=77.90.31.10:4519 , piece_id=YZBM7ZVYOHTBDUFMFPVOYDWSYII5IYJOVNFZAT5VJ24OWAQLJYIA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.634717239 (636h13m55.505800284s) [noise=true , address=145.79.227.14:4839 , piece_id=HNZBYCAFHIBJEU2PV2MUZ2FGE3O2BHTIX4LXMSUAVSKKICCPKDZQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.638540770 (636h13m55.509623815s) [noise=true , address=77.90.31.12:4589 , piece_id=7ERRHJAUHFPWAI5CSBHSMIBV62BNGGTAXIBETXCYGDNP23VLDKCQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.640554774 (636h13m55.511637819s) [noise=true , address=45.91.116.8:4046 , piece_id=VGW2MI5LCOY5M724R6FKXXYWX3ZUZWGRJAP4OAUCZEGGKGE6JWZQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:57.330501460 (636h13m57.201584496s) [upload=557491]: Next returned nil segment Jan 1 03:42:57.330514725 (636h13m57.20159776s) [upload=557491]: Waiting for error group managing segments... Jan 1 03:42:57.609560712 (636h13m57.480643747s) [noise=true , address=145.79.227.36:4519 , piece_id=H5SGXIA3YRSEKQURYPTVRJ5FY2BDSSBYFVJNYUSL2VDRGBOWQPEQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.620460857 (636h13m57.491543892s) [noise=true , address=145.79.227.20:4439 , piece_id=GW5FHVJDWNUB72SEFIZ4L77K4OWY5T7IAHE3HCCWRLS2BSRAX2KQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.621281650 (636h13m57.492364685s) [noise=true , address=145.79.227.15:4875 , piece_id=Y5FT7KRMZRSCLVOSQE3MJPIMQK6CJMGNQNNSWFHINYPGD2WAACHA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.626638129 (636h13m57.497721174s) [noise=true , address=145.79.227.51:4411 , piece_id=XDHEPJUSHDRYEAABUXO7P3SX6CTJCN3WFE6CEVS4MBREC3PZGZMQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.634713907 (636h13m57.505796952s) [noise=true , address=145.79.227.4:4187 , piece_id=QKE5QSIHAFU7QHQAGXBRWHDPQB6OID5X46NQKLCKE666XBRURUYA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.634782467 (636h13m57.505865512s) [noise=true , address=145.79.227.17:4970 , piece_id=KM4GYXML3JJ3XKA4WW4UQ67XQ7VUR64YWMFDHZB2YZXJRNNVKF3Q, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.635410192 (636h13m57.506493237s) [noise=true , address=145.79.227.19:4077 , piece_id=3QYQIMFOMTFCXVRV2OPQZHYCV7SYXLPF3B2U4H56NGYJ7LCQDHXQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.646671215 (636h13m57.51775426s) [noise=true , address=145.79.227.39:4013 , piece_id=CW4ZOVRJLSB2ORCD3EC7TPFGSTMM3GTMQI45QOM4VTKTAV6OKVYA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.724274681 (636h13m57.595357716s) [noise=true , address=145.79.227.14:4839 , piece_id=HNZBYCAFHIBJEU2PV2MUZ2FGE3O2BHTIX4LXMSUAVSKKICCPKDZQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.877926331 (636h13m57.749009376s) [noise=true , address=77.90.31.21:4878 , piece_id=FBLNPCSXSNK4JGPJ326ER7T6ZOFND33MYWQRWESSTTRI7HILXQOA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.923048219 (636h13m57.794131304s) [noise=true , address=77.90.31.19:4842 , piece_id=HNKE2G6U56TH7IRXOEQVDDK2AGJFP2JXMM6PRLEL6ZFUP757EDVQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.936788983 (636h13m57.807872018s) [noise=true , address=45.91.116.12:4226 , piece_id=DFMRPG3SR4XVT6DLMHUSKZ7ZKOJZ2NV7MAEPVQ5QRPKFSXV3ZLHA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.947857278 (636h13m57.818940313s) [noise=true , address=45.91.116.10:4081 , piece_id=7CO63CDAFPC6EKDNMGXMTSCUFJFX7H4XFLKFSL3M25D3ET4OUDWQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.947876315 (636h13m57.81895936s) [seg_pos={0 0}, upload=557491]: Segment reached optimal threshold of 13 pieces. Jan 1 03:42:57.948102415 (636h13m57.81918545s) [noise=true , address=77.90.31.10:4519 , piece_id=YZBM7ZVYOHTBDUFMFPVOYDWSYII5IYJOVNFZAT5VJ24OWAQLJYIA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 1 03:42:57.948110320 (636h13m57.819193355s) [noise=true , address=77.90.31.10:4519 , piece_id=YZBM7ZVYOHTBDUFMFPVOYDWSYII5IYJOVNFZAT5VJ24OWAQLJYIA, seg_pos={0 0}, upload=557491]: Long tail cancellation due to optimal threshold Jan 1 03:42:57.948337112 (636h13m57.819420147s) [noise=true , address=45.91.116.8:4046 , piece_id=VGW2MI5LCOY5M724R6FKXXYWX3ZUZWGRJAP4OAUCZEGGKGE6JWZQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Jan 1 03:42:57.948343814 (636h13m57.81942685s) [noise=true , address=45.91.116.8:4046 , piece_id=VGW2MI5LCOY5M724R6FKXXYWX3ZUZWGRJAP4OAUCZEGGKGE6JWZQ, seg_pos={0 0}, upload=557491]: Long tail cancellation due to optimal threshold Jan 1 03:42:57.948452912 (636h13m57.819535957s) [noise=true , address=77.90.31.12:4589 , piece_id=7ERRHJAUHFPWAI5CSBHSMIBV62BNGGTAXIBETXCYGDNP23VLDKCQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jan 1 03:42:57.948496194 (636h13m57.819579229s) [noise=true , address=77.90.31.12:4589 , piece_id=7ERRHJAUHFPWAI5CSBHSMIBV62BNGGTAXIBETXCYGDNP23VLDKCQ, seg_pos={0 0}, upload=557491]: Long tail cancellation due to optimal threshold Jan 1 03:42:57.948530439 (636h13m57.819613475s) [seg_pos={0 0}, upload=557491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:42:57.948649797 (636h13m57.819732842s) [upload=557491]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:42:57.948657161 (636h13m57.819740196s) [upload=557491]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:42:58.242849077 (636h13m58.113932132s) [upload=557491]: Done uploading segments... Jan 1 03:42:58.242870609 (636h13m58.113953644s) [upload=557491]: Upload finished. err: <nil> Duration: 6.097111073s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [........X......X............................X........X..X...] duration: 3.383758549s trace: https://jaeger.dev.storj.io/trace/446828087de3e02d FAIL (2026-01-01 03:43:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.383790149s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X....X........................................X....] duration: 38.393900513s trace: https://jaeger.dev.storj.io/trace/1ebdd84a47b0fea8 FAIL (2026-01-01 03:43:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555732730 (636h10m14.426815775s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555861985 (636h10m14.42694503s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.556167417 (636h10m14.427250462s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557132745 (636h10m14.42821578s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557290145 (636h10m14.42837318s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.558546016 (636h10m14.429629051s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.559810844 (636h10m14.430893889s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.561308566 (636h10m14.432391601s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.562571591 (636h10m14.433654636s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.563473288 (636h10m14.434556333s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.564781790 (636h10m14.435864825s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.565835766 (636h10m14.436918801s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567015342 (636h10m14.438098387s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567628530 (636h10m14.438711565s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:16.014065836 (636h10m15.885148871s) [upload=557441]: Next returned nil segment Jan 1 03:39:16.014083580 (636h10m15.885166615s) [upload=557441]: Waiting for error group managing segments... Jan 1 03:39:16.455400629 (636h10m16.326483664s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.464872484 (636h10m16.335955529s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.616875776 (636h10m16.487958821s) [noise=true , address=77.90.31.19:4876 , piece_id=UVRE5SNVM7SVUWD37QXCFDL7FWRQIMGFBYKLUULJKQJ4SW6224PA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.759240917 (636h10m16.630323952s) [noise=true , address=77.90.31.21:4907 , piece_id=XELNGQJ3EHMREVTXIFRR4FLXDD5U6CMAQ53GRSJ4WOT76QOQCBGQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.826797803 (636h10m16.697880848s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.039826124 (636h10m16.910909169s) [noise=true , address=145.79.227.49:4357 , piece_id=NRGFOQVKNCENLBXQDAU63ZQTJXYDJZ5YMEMZUHNBAWCERZCDI5EQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.319810887 (636h10m17.190893922s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.335275864 (636h10m17.206358899s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.360870549 (636h10m17.231953594s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.412439583 (636h10m17.283522628s) [noise=true , address=145.79.227.35:4513 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.469220361 (636h10m17.340303396s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.485589420 (636h10m17.356672455s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.598713200 (636h10m17.469796245s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911962313 (636h10m17.783045348s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911978394 (636h10m17.783061429s) [seg_pos={0 0}, upload=557441]: Segment reached optimal threshold of 14 pieces. Jan 1 03:39:17.912112600 (636h10m17.783195635s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912118220 (636h10m17.783201255s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912348508 (636h10m17.783431543s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912354901 (636h10m17.783437936s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912458238 (636h10m17.783541283s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection Jan 1 03:39:17.912463217 (636h10m17.783546252s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912551856 (636h10m17.783634891s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE): protocol: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection Jan 1 03:39:17.912557115 (636h10m17.783640151s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912582464 (636h10m17.783665499s) [seg_pos={0 0}, upload=557441]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:39:17.912645484 (636h10m17.783728519s) [upload=557441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:39:17.912649532 (636h10m17.783732567s) [upload=557441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:39:18.208729342 (636h10m18.079812378s) [upload=557441]: Done uploading segments... Jan 1 03:39:18.208744913 (636h10m18.079827958s) [upload=557441]: Upload finished. err: <nil> Duration: 38.393921603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.........X......X............................X........X..X..] duration: 3.542590097s trace: https://jaeger.dev.storj.io/trace/446828087de3e02d FAIL (2026-01-01 03:43:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542623119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [..X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX...X.X] duration: 30.003481005s trace: https://jaeger.dev.storj.io/trace/1c54f72d9490089d FAIL (2026-01-01 03:43:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: FBLNPCSXSNK4JGPJ326ER7T6ZOFND33MYWQRWESSTTRI7HILXQOA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 7CO63CDAFPC6EKDNMGXMTSCUFJFX7H4XFLKFSL3M25D3ET4OUDWQ) context deadline exceeded ======LOGS====== Jan 1 03:42:55.370396032 (636h13m55.241479077s) [upload=557491]: Starting upload Jan 1 03:42:55.370481203 (636h13m55.241564248s) [upload=557491]: Done starting upload Jan 1 03:42:55.370558051 (636h13m55.241641086s) [upload=557491]: Uploading segments... Jan 1 03:42:55.370727773 (636h13m55.241810818s) [upload=557491]: Got next segment. Inline: false Jan 1 03:42:55.370768892 (636h13m55.241851927s) [upload=557491]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:42:55.370774262 (636h13m55.241857297s) [upload=557491]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:42:55.615244972 (636h13m55.486328037s) [seg_pos={0 0}, upload=557491]: Begin upload segment... Jan 1 03:42:55.615319634 (636h13m55.486402669s) [seg_pos={0 0}, upload=557491]: Done begin upload segment. Jan 1 03:42:55.615442186 (636h13m55.486525222s) [noise=true , address=145.79.227.19:4077 , piece_id=3QYQIMFOMTFCXVRV2OPQZHYCV7SYXLPF3B2U4H56NGYJ7LCQDHXQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.615495849 (636h13m55.486578904s) [noise=true , address=145.79.227.36:4519 , piece_id=H5SGXIA3YRSEKQURYPTVRJ5FY2BDSSBYFVJNYUSL2VDRGBOWQPEQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.616143773 (636h13m55.487226818s) [noise=true , address=145.79.227.17:4970 , piece_id=KM4GYXML3JJ3XKA4WW4UQ67XQ7VUR64YWMFDHZB2YZXJRNNVKF3Q, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.616856400 (636h13m55.487939435s) [noise=true , address=145.79.227.4:4187 , piece_id=QKE5QSIHAFU7QHQAGXBRWHDPQB6OID5X46NQKLCKE666XBRURUYA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.617558527 (636h13m55.488641562s) [noise=true , address=77.90.31.21:4878 , piece_id=FBLNPCSXSNK4JGPJ326ER7T6ZOFND33MYWQRWESSTTRI7HILXQOA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.618241337 (636h13m55.489324382s) [noise=true , address=145.79.227.15:4875 , piece_id=Y5FT7KRMZRSCLVOSQE3MJPIMQK6CJMGNQNNSWFHINYPGD2WAACHA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.618847992 (636h13m55.489931037s) [noise=true , address=45.91.116.12:4226 , piece_id=DFMRPG3SR4XVT6DLMHUSKZ7ZKOJZ2NV7MAEPVQ5QRPKFSXV3ZLHA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.619355048 (636h13m55.490438093s) [noise=true , address=145.79.227.51:4411 , piece_id=XDHEPJUSHDRYEAABUXO7P3SX6CTJCN3WFE6CEVS4MBREC3PZGZMQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.620738171 (636h13m55.491821246s) [noise=true , address=145.79.227.39:4013 , piece_id=CW4ZOVRJLSB2ORCD3EC7TPFGSTMM3GTMQI45QOM4VTKTAV6OKVYA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.620906642 (636h13m55.491989677s) [noise=true , address=77.90.31.19:4842 , piece_id=HNKE2G6U56TH7IRXOEQVDDK2AGJFP2JXMM6PRLEL6ZFUP757EDVQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.622842278 (636h13m55.493925313s) [noise=true , address=45.91.116.10:4081 , piece_id=7CO63CDAFPC6EKDNMGXMTSCUFJFX7H4XFLKFSL3M25D3ET4OUDWQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.623066144 (636h13m55.494149179s) [noise=true , address=145.79.227.20:4439 , piece_id=GW5FHVJDWNUB72SEFIZ4L77K4OWY5T7IAHE3HCCWRLS2BSRAX2KQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.624645010 (636h13m55.495728055s) [noise=true , address=77.90.31.10:4519 , piece_id=YZBM7ZVYOHTBDUFMFPVOYDWSYII5IYJOVNFZAT5VJ24OWAQLJYIA, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.634717239 (636h13m55.505800284s) [noise=true , address=145.79.227.14:4839 , piece_id=HNZBYCAFHIBJEU2PV2MUZ2FGE3O2BHTIX4LXMSUAVSKKICCPKDZQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.638540770 (636h13m55.509623815s) [noise=true , address=77.90.31.12:4589 , piece_id=7ERRHJAUHFPWAI5CSBHSMIBV62BNGGTAXIBETXCYGDNP23VLDKCQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:55.640554774 (636h13m55.511637819s) [noise=true , address=45.91.116.8:4046 , piece_id=VGW2MI5LCOY5M724R6FKXXYWX3ZUZWGRJAP4OAUCZEGGKGE6JWZQ, seg_pos={0 0}, upload=557491]: Uploading piece... Jan 1 03:42:57.330501460 (636h13m57.201584496s) [upload=557491]: Next returned nil segment Jan 1 03:42:57.330514725 (636h13m57.20159776s) [upload=557491]: Waiting for error group managing segments... Jan 1 03:42:57.609560712 (636h13m57.480643747s) [noise=true , address=145.79.227.36:4519 , piece_id=H5SGXIA3YRSEKQURYPTVRJ5FY2BDSSBYFVJNYUSL2VDRGBOWQPEQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.620460857 (636h13m57.491543892s) [noise=true , address=145.79.227.20:4439 , piece_id=GW5FHVJDWNUB72SEFIZ4L77K4OWY5T7IAHE3HCCWRLS2BSRAX2KQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.621281650 (636h13m57.492364685s) [noise=true , address=145.79.227.15:4875 , piece_id=Y5FT7KRMZRSCLVOSQE3MJPIMQK6CJMGNQNNSWFHINYPGD2WAACHA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.626638129 (636h13m57.497721174s) [noise=true , address=145.79.227.51:4411 , piece_id=XDHEPJUSHDRYEAABUXO7P3SX6CTJCN3WFE6CEVS4MBREC3PZGZMQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.634713907 (636h13m57.505796952s) [noise=true , address=145.79.227.4:4187 , piece_id=QKE5QSIHAFU7QHQAGXBRWHDPQB6OID5X46NQKLCKE666XBRURUYA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.634782467 (636h13m57.505865512s) [noise=true , address=145.79.227.17:4970 , piece_id=KM4GYXML3JJ3XKA4WW4UQ67XQ7VUR64YWMFDHZB2YZXJRNNVKF3Q, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.635410192 (636h13m57.506493237s) [noise=true , address=145.79.227.19:4077 , piece_id=3QYQIMFOMTFCXVRV2OPQZHYCV7SYXLPF3B2U4H56NGYJ7LCQDHXQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.646671215 (636h13m57.51775426s) [noise=true , address=145.79.227.39:4013 , piece_id=CW4ZOVRJLSB2ORCD3EC7TPFGSTMM3GTMQI45QOM4VTKTAV6OKVYA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.724274681 (636h13m57.595357716s) [noise=true , address=145.79.227.14:4839 , piece_id=HNZBYCAFHIBJEU2PV2MUZ2FGE3O2BHTIX4LXMSUAVSKKICCPKDZQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.877926331 (636h13m57.749009376s) [noise=true , address=77.90.31.21:4878 , piece_id=FBLNPCSXSNK4JGPJ326ER7T6ZOFND33MYWQRWESSTTRI7HILXQOA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.923048219 (636h13m57.794131304s) [noise=true , address=77.90.31.19:4842 , piece_id=HNKE2G6U56TH7IRXOEQVDDK2AGJFP2JXMM6PRLEL6ZFUP757EDVQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.936788983 (636h13m57.807872018s) [noise=true , address=45.91.116.12:4226 , piece_id=DFMRPG3SR4XVT6DLMHUSKZ7ZKOJZ2NV7MAEPVQ5QRPKFSXV3ZLHA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.947857278 (636h13m57.818940313s) [noise=true , address=45.91.116.10:4081 , piece_id=7CO63CDAFPC6EKDNMGXMTSCUFJFX7H4XFLKFSL3M25D3ET4OUDWQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: <nil> Jan 1 03:42:57.947876315 (636h13m57.81895936s) [seg_pos={0 0}, upload=557491]: Segment reached optimal threshold of 13 pieces. Jan 1 03:42:57.948102415 (636h13m57.81918545s) [noise=true , address=77.90.31.10:4519 , piece_id=YZBM7ZVYOHTBDUFMFPVOYDWSYII5IYJOVNFZAT5VJ24OWAQLJYIA, seg_pos={0 0}, upload=557491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 1 03:42:57.948110320 (636h13m57.819193355s) [noise=true , address=77.90.31.10:4519 , piece_id=YZBM7ZVYOHTBDUFMFPVOYDWSYII5IYJOVNFZAT5VJ24OWAQLJYIA, seg_pos={0 0}, upload=557491]: Long tail cancellation due to optimal threshold Jan 1 03:42:57.948337112 (636h13m57.819420147s) [noise=true , address=45.91.116.8:4046 , piece_id=VGW2MI5LCOY5M724R6FKXXYWX3ZUZWGRJAP4OAUCZEGGKGE6JWZQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Jan 1 03:42:57.948343814 (636h13m57.81942685s) [noise=true , address=45.91.116.8:4046 , piece_id=VGW2MI5LCOY5M724R6FKXXYWX3ZUZWGRJAP4OAUCZEGGKGE6JWZQ, seg_pos={0 0}, upload=557491]: Long tail cancellation due to optimal threshold Jan 1 03:42:57.948452912 (636h13m57.819535957s) [noise=true , address=77.90.31.12:4589 , piece_id=7ERRHJAUHFPWAI5CSBHSMIBV62BNGGTAXIBETXCYGDNP23VLDKCQ, seg_pos={0 0}, upload=557491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Jan 1 03:42:57.948496194 (636h13m57.819579229s) [noise=true , address=77.90.31.12:4589 , piece_id=7ERRHJAUHFPWAI5CSBHSMIBV62BNGGTAXIBETXCYGDNP23VLDKCQ, seg_pos={0 0}, upload=557491]: Long tail cancellation due to optimal threshold Jan 1 03:42:57.948530439 (636h13m57.819613475s) [seg_pos={0 0}, upload=557491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:42:57.948649797 (636h13m57.819732842s) [upload=557491]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:42:57.948657161 (636h13m57.819740196s) [upload=557491]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:42:58.242849077 (636h13m58.113932132s) [upload=557491]: Done uploading segments... Jan 1 03:42:58.242870609 (636h13m58.113953644s) [upload=557491]: Upload finished. err: <nil> Duration: 30.003508246s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [..........X......X............................X........X..X.] duration: 4.539827614s trace: https://jaeger.dev.storj.io/trace/446828087de3e02d FAIL (2026-01-01 03:43:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.539841721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X....X........................................X...] duration: 28.711926729s trace: https://jaeger.dev.storj.io/trace/1ebdd84a47b0fea8 FAIL (2026-01-01 03:42:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555732730 (636h10m14.426815775s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555861985 (636h10m14.42694503s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.556167417 (636h10m14.427250462s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557132745 (636h10m14.42821578s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557290145 (636h10m14.42837318s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.558546016 (636h10m14.429629051s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.559810844 (636h10m14.430893889s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.561308566 (636h10m14.432391601s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.562571591 (636h10m14.433654636s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.563473288 (636h10m14.434556333s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.564781790 (636h10m14.435864825s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.565835766 (636h10m14.436918801s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567015342 (636h10m14.438098387s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567628530 (636h10m14.438711565s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:16.014065836 (636h10m15.885148871s) [upload=557441]: Next returned nil segment Jan 1 03:39:16.014083580 (636h10m15.885166615s) [upload=557441]: Waiting for error group managing segments... Jan 1 03:39:16.455400629 (636h10m16.326483664s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.464872484 (636h10m16.335955529s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.616875776 (636h10m16.487958821s) [noise=true , address=77.90.31.19:4876 , piece_id=UVRE5SNVM7SVUWD37QXCFDL7FWRQIMGFBYKLUULJKQJ4SW6224PA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.759240917 (636h10m16.630323952s) [noise=true , address=77.90.31.21:4907 , piece_id=XELNGQJ3EHMREVTXIFRR4FLXDD5U6CMAQ53GRSJ4WOT76QOQCBGQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.826797803 (636h10m16.697880848s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.039826124 (636h10m16.910909169s) [noise=true , address=145.79.227.49:4357 , piece_id=NRGFOQVKNCENLBXQDAU63ZQTJXYDJZ5YMEMZUHNBAWCERZCDI5EQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.319810887 (636h10m17.190893922s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.335275864 (636h10m17.206358899s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.360870549 (636h10m17.231953594s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.412439583 (636h10m17.283522628s) [noise=true , address=145.79.227.35:4513 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.469220361 (636h10m17.340303396s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.485589420 (636h10m17.356672455s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.598713200 (636h10m17.469796245s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911962313 (636h10m17.783045348s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911978394 (636h10m17.783061429s) [seg_pos={0 0}, upload=557441]: Segment reached optimal threshold of 14 pieces. Jan 1 03:39:17.912112600 (636h10m17.783195635s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912118220 (636h10m17.783201255s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912348508 (636h10m17.783431543s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912354901 (636h10m17.783437936s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912458238 (636h10m17.783541283s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection Jan 1 03:39:17.912463217 (636h10m17.783546252s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912551856 (636h10m17.783634891s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE): protocol: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection Jan 1 03:39:17.912557115 (636h10m17.783640151s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912582464 (636h10m17.783665499s) [seg_pos={0 0}, upload=557441]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:39:17.912645484 (636h10m17.783728519s) [upload=557441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:39:17.912649532 (636h10m17.783732567s) [upload=557441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:39:18.208729342 (636h10m18.079812378s) [upload=557441]: Done uploading segments... Jan 1 03:39:18.208744913 (636h10m18.079827958s) [upload=557441]: Upload finished. err: <nil> Duration: 28.711951526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [...........X......X............................X........X..X] duration: 5.001327522s trace: https://jaeger.dev.storj.io/trace/446828087de3e02d FAIL (2026-01-01 03:42:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001351117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX...X.] duration: 6.50818377s trace: https://jaeger.dev.storj.io/trace/578e2d38a7451922 FAIL (2026-01-01 03:42:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:41:23.760805468 (636h12m23.631888503s) [upload=557472]: Starting upload Jan 1 03:41:23.760931168 (636h12m23.632014203s) [upload=557472]: Done starting upload Jan 1 03:41:23.761059753 (636h12m23.632142788s) [upload=557472]: Uploading segments... Jan 1 03:41:23.761244134 (636h12m23.632327179s) [upload=557472]: Got next segment. Inline: false Jan 1 03:41:23.761289019 (636h12m23.632372064s) [upload=557472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:41:23.761294679 (636h12m23.632377725s) [upload=557472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:41:24.005225724 (636h12m23.876308759s) [seg_pos={0 0}, upload=557472]: Begin upload segment... Jan 1 03:41:24.005265580 (636h12m23.876348625s) [seg_pos={0 0}, upload=557472]: Done begin upload segment. Jan 1 03:41:24.005302119 (636h12m23.876385154s) [noise=true , address=145.79.227.44:4193 , piece_id=37VO3UL7VLUUPDQX77IZSQRFIKUD5PWXCCLTIDZHW46NQ3DHNBQA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.005319352 (636h12m23.876402387s) [noise=true , address=145.79.227.6:4258 , piece_id=SMO2K34YK4DVDZGLROCBJX6JYMZNEEFSISWIMOWR6CTN5A3467EQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.005947307 (636h12m23.877030352s) [noise=true , address=145.79.227.12:4765 , piece_id=CJBYBAU2IOR2PMBSVZE5OPYJKMBKJVZTSU7FIFIR736PGS5KWZWQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.005958199 (636h12m23.877041244s) [noise=true , address=145.79.227.38:4977 , piece_id=26ZRSSF4XZEESUMZB5TE4PSRM3BWT27LZJN7LEKDVMWRQMGYF43Q, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006293477 (636h12m23.877376512s) [noise=true , address=145.79.227.8:4330 , piece_id=VFJVHZGG5IADJ3QOPENHZ3BS3E7225IEBK3J3JDY6GK7FBJDJWKA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006383919 (636h12m23.877466964s) [noise=true , address=145.79.227.37:4566 , piece_id=7VAGGFQZC76L7OW2RKJ6ICX3SHGTDEJT42XPYXUGWOQM6X6E2SEQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006580674 (636h12m23.877663719s) [noise=true , address=145.79.227.11:4730 , piece_id=MSTE7YQULV46R7GZGHXHOP7BZECL5RHVILPDPJXG6CWRYDBWBUHQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006798148 (636h12m23.877881193s) [noise=true , address=45.91.116.7:4010 , piece_id=55VM6V4D3TO7FMWJDPDERFFWGHYOEEEDCM3DU5SER46RHC3EKR6Q, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006928547 (636h12m23.878011592s) [noise=true , address=145.79.227.3:4151 , piece_id=WUVJJQEZXID5SOCQ2H264P5I2K2LWK6QFTAOU6SZZJ7YHI2QQBPA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007226243 (636h12m23.878309288s) [noise=true , address=145.79.227.51:4410 , piece_id=SLNW7BQVL7O3NQJXZ3LR737T76WIZKEIIGIWW5EY7BIASW5OTLMA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007241121 (636h12m23.878324186s) [noise=true , address=77.90.31.17:4773 , piece_id=RQAKIN7NOO2OBS7Z4KAJPWPOY6INH7RAHMESSLMS5PVC4YX7EK6A, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007629291 (636h12m23.878712336s) [noise=true , address=77.90.31.12:4590 , piece_id=M4VFD4RD467ZP7DXPOM76DL37ET6QTXGLYAMIMJPMFBZEGCRQG6Q, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007771181 (636h12m23.878854226s) [noise=true , address=145.79.227.27:4689 , piece_id=ECCFQVZDR2VD2XYLQPPSCSTFTJP3IPHFKCIV2FWPOHIUXU7DATSQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.008011939 (636h12m23.879094974s) [noise=true , address=45.91.116.6:4976 , piece_id=Y2CH7CJ6S7B2TJWBMRFE5R3BQXTLN5ULEKTI7CVFDARQXU72L5NQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.008198785 (636h12m23.87928183s) [noise=true , address=77.90.31.10:4517 , piece_id=A5KUMBH7BWYGXF744BTA4ZEH4A6ZIUKBOJSVGKVOQM6HPRT2JU6A, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.009086705 (636h12m23.88016975s) [noise=true , address=45.91.116.14:4189 , piece_id=OO5UWBNKAM2JR2HRCBKCOARVBGAQGJYDF375KFZ4AWHSB6E3QOKQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:25.709979115 (636h12m25.58106216s) [upload=557472]: Next returned nil segment Jan 1 03:41:25.709996629 (636h12m25.581079664s) [upload=557472]: Waiting for error group managing segments... Jan 1 03:41:25.996252633 (636h12m25.867335678s) [noise=true , address=145.79.227.38:4977 , piece_id=26ZRSSF4XZEESUMZB5TE4PSRM3BWT27LZJN7LEKDVMWRQMGYF43Q, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:25.996528799 (636h12m25.867611844s) [noise=true , address=145.79.227.6:4258 , piece_id=SMO2K34YK4DVDZGLROCBJX6JYMZNEEFSISWIMOWR6CTN5A3467EQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:25.998327114 (636h12m25.869410159s) [noise=true , address=145.79.227.3:4151 , piece_id=WUVJJQEZXID5SOCQ2H264P5I2K2LWK6QFTAOU6SZZJ7YHI2QQBPA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.001891932 (636h12m25.872974967s) [noise=true , address=145.79.227.37:4566 , piece_id=7VAGGFQZC76L7OW2RKJ6ICX3SHGTDEJT42XPYXUGWOQM6X6E2SEQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.003898122 (636h12m25.874981167s) [noise=true , address=145.79.227.51:4410 , piece_id=SLNW7BQVL7O3NQJXZ3LR737T76WIZKEIIGIWW5EY7BIASW5OTLMA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.010471007 (636h12m25.881554052s) [noise=true , address=145.79.227.12:4765 , piece_id=CJBYBAU2IOR2PMBSVZE5OPYJKMBKJVZTSU7FIFIR736PGS5KWZWQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.029444585 (636h12m25.90052762s) [noise=true , address=145.79.227.11:4730 , piece_id=MSTE7YQULV46R7GZGHXHOP7BZECL5RHVILPDPJXG6CWRYDBWBUHQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.035855903 (636h12m25.906938938s) [noise=true , address=145.79.227.8:4330 , piece_id=VFJVHZGG5IADJ3QOPENHZ3BS3E7225IEBK3J3JDY6GK7FBJDJWKA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.172690202 (636h12m26.043773247s) [noise=true , address=145.79.227.27:4689 , piece_id=ECCFQVZDR2VD2XYLQPPSCSTFTJP3IPHFKCIV2FWPOHIUXU7DATSQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.299178520 (636h12m26.170261565s) [noise=true , address=45.91.116.6:4976 , piece_id=Y2CH7CJ6S7B2TJWBMRFE5R3BQXTLN5ULEKTI7CVFDARQXU72L5NQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.348111114 (636h12m26.219194149s) [noise=true , address=45.91.116.7:4010 , piece_id=55VM6V4D3TO7FMWJDPDERFFWGHYOEEEDCM3DU5SER46RHC3EKR6Q, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.428664436 (636h12m26.299747471s) [noise=true , address=45.91.116.14:4189 , piece_id=OO5UWBNKAM2JR2HRCBKCOARVBGAQGJYDF375KFZ4AWHSB6E3QOKQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.871462616 (636h12m26.742545651s) [noise=true , address=77.90.31.10:4517 , piece_id=A5KUMBH7BWYGXF744BTA4ZEH4A6ZIUKBOJSVGKVOQM6HPRT2JU6A, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.871483817 (636h12m26.742566852s) [seg_pos={0 0}, upload=557472]: Segment reached optimal threshold of 13 pieces. Jan 1 03:41:26.871654852 (636h12m26.742737887s) [noise=true , address=145.79.227.44:4193 , piece_id=37VO3UL7VLUUPDQX77IZSQRFIKUD5PWXCCLTIDZHW46NQ3DHNBQA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Jan 1 03:41:26.871661876 (636h12m26.742744911s) [noise=true , address=145.79.227.44:4193 , piece_id=37VO3UL7VLUUPDQX77IZSQRFIKUD5PWXCCLTIDZHW46NQ3DHNBQA, seg_pos={0 0}, upload=557472]: Long tail cancellation due to optimal threshold Jan 1 03:41:26.871854993 (636h12m26.742938028s) [noise=true , address=77.90.31.12:4590 , piece_id=M4VFD4RD467ZP7DXPOM76DL37ET6QTXGLYAMIMJPMFBZEGCRQG6Q, seg_pos={0 0}, upload=557472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Jan 1 03:41:26.871883688 (636h12m26.742966733s) [noise=true , address=77.90.31.12:4590 , piece_id=M4VFD4RD467ZP7DXPOM76DL37ET6QTXGLYAMIMJPMFBZEGCRQG6Q, seg_pos={0 0}, upload=557472]: Long tail cancellation due to optimal threshold Jan 1 03:41:26.871895791 (636h12m26.742978826s) [noise=true , address=77.90.31.17:4773 , piece_id=RQAKIN7NOO2OBS7Z4KAJPWPOY6INH7RAHMESSLMS5PVC4YX7EK6A, seg_pos={0 0}, upload=557472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Jan 1 03:41:26.871904658 (636h12m26.742987694s) [noise=true , address=77.90.31.17:4773 , piece_id=RQAKIN7NOO2OBS7Z4KAJPWPOY6INH7RAHMESSLMS5PVC4YX7EK6A, seg_pos={0 0}, upload=557472]: Long tail cancellation due to optimal threshold Jan 1 03:41:26.871947390 (636h12m26.743030425s) [seg_pos={0 0}, upload=557472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:41:26.872014868 (636h12m26.743097903s) [upload=557472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:41:26.872020168 (636h12m26.743103203s) [upload=557472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:41:27.158644383 (636h12m27.029727418s) [upload=557472]: Done uploading segments... Jan 1 03:41:27.158663450 (636h12m27.029746485s) [upload=557472]: Upload finished. err: <nil> Duration: 6.508216803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....X........................................X..] duration: 37.613438468s trace: https://jaeger.dev.storj.io/trace/1ebdd84a47b0fea8 FAIL (2026-01-01 03:41:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555732730 (636h10m14.426815775s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555861985 (636h10m14.42694503s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.556167417 (636h10m14.427250462s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557132745 (636h10m14.42821578s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557290145 (636h10m14.42837318s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.558546016 (636h10m14.429629051s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.559810844 (636h10m14.430893889s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.561308566 (636h10m14.432391601s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.562571591 (636h10m14.433654636s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.563473288 (636h10m14.434556333s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.564781790 (636h10m14.435864825s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.565835766 (636h10m14.436918801s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567015342 (636h10m14.438098387s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567628530 (636h10m14.438711565s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:16.014065836 (636h10m15.885148871s) [upload=557441]: Next returned nil segment Jan 1 03:39:16.014083580 (636h10m15.885166615s) [upload=557441]: Waiting for error group managing segments... Jan 1 03:39:16.455400629 (636h10m16.326483664s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.464872484 (636h10m16.335955529s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.616875776 (636h10m16.487958821s) [noise=true , address=77.90.31.19:4876 , piece_id=UVRE5SNVM7SVUWD37QXCFDL7FWRQIMGFBYKLUULJKQJ4SW6224PA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.759240917 (636h10m16.630323952s) [noise=true , address=77.90.31.21:4907 , piece_id=XELNGQJ3EHMREVTXIFRR4FLXDD5U6CMAQ53GRSJ4WOT76QOQCBGQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.826797803 (636h10m16.697880848s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.039826124 (636h10m16.910909169s) [noise=true , address=145.79.227.49:4357 , piece_id=NRGFOQVKNCENLBXQDAU63ZQTJXYDJZ5YMEMZUHNBAWCERZCDI5EQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.319810887 (636h10m17.190893922s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.335275864 (636h10m17.206358899s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.360870549 (636h10m17.231953594s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.412439583 (636h10m17.283522628s) [noise=true , address=145.79.227.35:4513 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.469220361 (636h10m17.340303396s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.485589420 (636h10m17.356672455s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.598713200 (636h10m17.469796245s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911962313 (636h10m17.783045348s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911978394 (636h10m17.783061429s) [seg_pos={0 0}, upload=557441]: Segment reached optimal threshold of 14 pieces. Jan 1 03:39:17.912112600 (636h10m17.783195635s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912118220 (636h10m17.783201255s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912348508 (636h10m17.783431543s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912354901 (636h10m17.783437936s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912458238 (636h10m17.783541283s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection Jan 1 03:39:17.912463217 (636h10m17.783546252s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912551856 (636h10m17.783634891s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE): protocol: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection Jan 1 03:39:17.912557115 (636h10m17.783640151s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912582464 (636h10m17.783665499s) [seg_pos={0 0}, upload=557441]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:39:17.912645484 (636h10m17.783728519s) [upload=557441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:39:17.912649532 (636h10m17.783732567s) [upload=557441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:39:18.208729342 (636h10m18.079812378s) [upload=557441]: Done uploading segments... Jan 1 03:39:18.208744913 (636h10m18.079827958s) [upload=557441]: Upload finished. err: <nil> Duration: 37.613462443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX...X] duration: 30.003565291s trace: https://jaeger.dev.storj.io/trace/578e2d38a7451922 FAIL (2026-01-01 03:41:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:41:23.760805468 (636h12m23.631888503s) [upload=557472]: Starting upload Jan 1 03:41:23.760931168 (636h12m23.632014203s) [upload=557472]: Done starting upload Jan 1 03:41:23.761059753 (636h12m23.632142788s) [upload=557472]: Uploading segments... Jan 1 03:41:23.761244134 (636h12m23.632327179s) [upload=557472]: Got next segment. Inline: false Jan 1 03:41:23.761289019 (636h12m23.632372064s) [upload=557472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:41:23.761294679 (636h12m23.632377725s) [upload=557472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:41:24.005225724 (636h12m23.876308759s) [seg_pos={0 0}, upload=557472]: Begin upload segment... Jan 1 03:41:24.005265580 (636h12m23.876348625s) [seg_pos={0 0}, upload=557472]: Done begin upload segment. Jan 1 03:41:24.005302119 (636h12m23.876385154s) [noise=true , address=145.79.227.44:4193 , piece_id=37VO3UL7VLUUPDQX77IZSQRFIKUD5PWXCCLTIDZHW46NQ3DHNBQA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.005319352 (636h12m23.876402387s) [noise=true , address=145.79.227.6:4258 , piece_id=SMO2K34YK4DVDZGLROCBJX6JYMZNEEFSISWIMOWR6CTN5A3467EQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.005947307 (636h12m23.877030352s) [noise=true , address=145.79.227.12:4765 , piece_id=CJBYBAU2IOR2PMBSVZE5OPYJKMBKJVZTSU7FIFIR736PGS5KWZWQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.005958199 (636h12m23.877041244s) [noise=true , address=145.79.227.38:4977 , piece_id=26ZRSSF4XZEESUMZB5TE4PSRM3BWT27LZJN7LEKDVMWRQMGYF43Q, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006293477 (636h12m23.877376512s) [noise=true , address=145.79.227.8:4330 , piece_id=VFJVHZGG5IADJ3QOPENHZ3BS3E7225IEBK3J3JDY6GK7FBJDJWKA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006383919 (636h12m23.877466964s) [noise=true , address=145.79.227.37:4566 , piece_id=7VAGGFQZC76L7OW2RKJ6ICX3SHGTDEJT42XPYXUGWOQM6X6E2SEQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006580674 (636h12m23.877663719s) [noise=true , address=145.79.227.11:4730 , piece_id=MSTE7YQULV46R7GZGHXHOP7BZECL5RHVILPDPJXG6CWRYDBWBUHQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006798148 (636h12m23.877881193s) [noise=true , address=45.91.116.7:4010 , piece_id=55VM6V4D3TO7FMWJDPDERFFWGHYOEEEDCM3DU5SER46RHC3EKR6Q, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006928547 (636h12m23.878011592s) [noise=true , address=145.79.227.3:4151 , piece_id=WUVJJQEZXID5SOCQ2H264P5I2K2LWK6QFTAOU6SZZJ7YHI2QQBPA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007226243 (636h12m23.878309288s) [noise=true , address=145.79.227.51:4410 , piece_id=SLNW7BQVL7O3NQJXZ3LR737T76WIZKEIIGIWW5EY7BIASW5OTLMA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007241121 (636h12m23.878324186s) [noise=true , address=77.90.31.17:4773 , piece_id=RQAKIN7NOO2OBS7Z4KAJPWPOY6INH7RAHMESSLMS5PVC4YX7EK6A, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007629291 (636h12m23.878712336s) [noise=true , address=77.90.31.12:4590 , piece_id=M4VFD4RD467ZP7DXPOM76DL37ET6QTXGLYAMIMJPMFBZEGCRQG6Q, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007771181 (636h12m23.878854226s) [noise=true , address=145.79.227.27:4689 , piece_id=ECCFQVZDR2VD2XYLQPPSCSTFTJP3IPHFKCIV2FWPOHIUXU7DATSQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.008011939 (636h12m23.879094974s) [noise=true , address=45.91.116.6:4976 , piece_id=Y2CH7CJ6S7B2TJWBMRFE5R3BQXTLN5ULEKTI7CVFDARQXU72L5NQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.008198785 (636h12m23.87928183s) [noise=true , address=77.90.31.10:4517 , piece_id=A5KUMBH7BWYGXF744BTA4ZEH4A6ZIUKBOJSVGKVOQM6HPRT2JU6A, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.009086705 (636h12m23.88016975s) [noise=true , address=45.91.116.14:4189 , piece_id=OO5UWBNKAM2JR2HRCBKCOARVBGAQGJYDF375KFZ4AWHSB6E3QOKQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:25.709979115 (636h12m25.58106216s) [upload=557472]: Next returned nil segment Jan 1 03:41:25.709996629 (636h12m25.581079664s) [upload=557472]: Waiting for error group managing segments... Jan 1 03:41:25.996252633 (636h12m25.867335678s) [noise=true , address=145.79.227.38:4977 , piece_id=26ZRSSF4XZEESUMZB5TE4PSRM3BWT27LZJN7LEKDVMWRQMGYF43Q, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:25.996528799 (636h12m25.867611844s) [noise=true , address=145.79.227.6:4258 , piece_id=SMO2K34YK4DVDZGLROCBJX6JYMZNEEFSISWIMOWR6CTN5A3467EQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:25.998327114 (636h12m25.869410159s) [noise=true , address=145.79.227.3:4151 , piece_id=WUVJJQEZXID5SOCQ2H264P5I2K2LWK6QFTAOU6SZZJ7YHI2QQBPA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.001891932 (636h12m25.872974967s) [noise=true , address=145.79.227.37:4566 , piece_id=7VAGGFQZC76L7OW2RKJ6ICX3SHGTDEJT42XPYXUGWOQM6X6E2SEQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.003898122 (636h12m25.874981167s) [noise=true , address=145.79.227.51:4410 , piece_id=SLNW7BQVL7O3NQJXZ3LR737T76WIZKEIIGIWW5EY7BIASW5OTLMA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.010471007 (636h12m25.881554052s) [noise=true , address=145.79.227.12:4765 , piece_id=CJBYBAU2IOR2PMBSVZE5OPYJKMBKJVZTSU7FIFIR736PGS5KWZWQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.029444585 (636h12m25.90052762s) [noise=true , address=145.79.227.11:4730 , piece_id=MSTE7YQULV46R7GZGHXHOP7BZECL5RHVILPDPJXG6CWRYDBWBUHQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.035855903 (636h12m25.906938938s) [noise=true , address=145.79.227.8:4330 , piece_id=VFJVHZGG5IADJ3QOPENHZ3BS3E7225IEBK3J3JDY6GK7FBJDJWKA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.172690202 (636h12m26.043773247s) [noise=true , address=145.79.227.27:4689 , piece_id=ECCFQVZDR2VD2XYLQPPSCSTFTJP3IPHFKCIV2FWPOHIUXU7DATSQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.299178520 (636h12m26.170261565s) [noise=true , address=45.91.116.6:4976 , piece_id=Y2CH7CJ6S7B2TJWBMRFE5R3BQXTLN5ULEKTI7CVFDARQXU72L5NQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.348111114 (636h12m26.219194149s) [noise=true , address=45.91.116.7:4010 , piece_id=55VM6V4D3TO7FMWJDPDERFFWGHYOEEEDCM3DU5SER46RHC3EKR6Q, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.428664436 (636h12m26.299747471s) [noise=true , address=45.91.116.14:4189 , piece_id=OO5UWBNKAM2JR2HRCBKCOARVBGAQGJYDF375KFZ4AWHSB6E3QOKQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.871462616 (636h12m26.742545651s) [noise=true , address=77.90.31.10:4517 , piece_id=A5KUMBH7BWYGXF744BTA4ZEH4A6ZIUKBOJSVGKVOQM6HPRT2JU6A, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.871483817 (636h12m26.742566852s) [seg_pos={0 0}, upload=557472]: Segment reached optimal threshold of 13 pieces. Jan 1 03:41:26.871654852 (636h12m26.742737887s) [noise=true , address=145.79.227.44:4193 , piece_id=37VO3UL7VLUUPDQX77IZSQRFIKUD5PWXCCLTIDZHW46NQ3DHNBQA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Jan 1 03:41:26.871661876 (636h12m26.742744911s) [noise=true , address=145.79.227.44:4193 , piece_id=37VO3UL7VLUUPDQX77IZSQRFIKUD5PWXCCLTIDZHW46NQ3DHNBQA, seg_pos={0 0}, upload=557472]: Long tail cancellation due to optimal threshold Jan 1 03:41:26.871854993 (636h12m26.742938028s) [noise=true , address=77.90.31.12:4590 , piece_id=M4VFD4RD467ZP7DXPOM76DL37ET6QTXGLYAMIMJPMFBZEGCRQG6Q, seg_pos={0 0}, upload=557472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Jan 1 03:41:26.871883688 (636h12m26.742966733s) [noise=true , address=77.90.31.12:4590 , piece_id=M4VFD4RD467ZP7DXPOM76DL37ET6QTXGLYAMIMJPMFBZEGCRQG6Q, seg_pos={0 0}, upload=557472]: Long tail cancellation due to optimal threshold Jan 1 03:41:26.871895791 (636h12m26.742978826s) [noise=true , address=77.90.31.17:4773 , piece_id=RQAKIN7NOO2OBS7Z4KAJPWPOY6INH7RAHMESSLMS5PVC4YX7EK6A, seg_pos={0 0}, upload=557472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Jan 1 03:41:26.871904658 (636h12m26.742987694s) [noise=true , address=77.90.31.17:4773 , piece_id=RQAKIN7NOO2OBS7Z4KAJPWPOY6INH7RAHMESSLMS5PVC4YX7EK6A, seg_pos={0 0}, upload=557472]: Long tail cancellation due to optimal threshold Jan 1 03:41:26.871947390 (636h12m26.743030425s) [seg_pos={0 0}, upload=557472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:41:26.872014868 (636h12m26.743097903s) [upload=557472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:41:26.872020168 (636h12m26.743103203s) [upload=557472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:41:27.158644383 (636h12m27.029727418s) [upload=557472]: Done uploading segments... Jan 1 03:41:27.158663450 (636h12m27.029746485s) [upload=557472]: Upload finished. err: <nil> Duration: 30.003582795s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X......X............................X........X..] duration: 3.569678078s trace: https://jaeger.dev.storj.io/trace/8b9520ee0ddc710 FAIL (2026-01-01 03:41:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569766146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX...] duration: 6.012484162s trace: https://jaeger.dev.storj.io/trace/3a1918a7502a0c40 FAIL (2026-01-01 03:41:07 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:38:18.274492318 (636h9m18.145575353s) [upload=557425]: Starting upload Jan 1 03:38:18.274567070 (636h9m18.145650105s) [upload=557425]: Done starting upload Jan 1 03:38:18.274688662 (636h9m18.145771707s) [upload=557425]: Uploading segments... Jan 1 03:38:18.274806416 (636h9m18.145889461s) [upload=557425]: Got next segment. Inline: false Jan 1 03:38:18.274842244 (636h9m18.145925289s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:38:18.274847464 (636h9m18.145930519s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:38:18.522138205 (636h9m18.39322127s) [seg_pos={0 0}, upload=557425]: Begin upload segment... Jan 1 03:38:18.522195464 (636h9m18.393278509s) [seg_pos={0 0}, upload=557425]: Done begin upload segment. Jan 1 03:38:18.522222756 (636h9m18.393305791s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522240590 (636h9m18.393323655s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522841954 (636h9m18.393924999s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523203123 (636h9m18.394286168s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523238200 (636h9m18.394321245s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523581053 (636h9m18.394664088s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523675452 (636h9m18.394758487s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523829435 (636h9m18.39491247s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523980252 (636h9m18.395063287s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524079642 (636h9m18.395162677s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524396465 (636h9m18.3954795s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524481868 (636h9m18.395564943s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524653404 (636h9m18.395736439s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524906376 (636h9m18.395989411s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524912417 (636h9m18.395995452s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.525774960 (636h9m18.396857995s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:20.226070182 (636h9m20.097153217s) [upload=557425]: Next returned nil segment Jan 1 03:38:20.226084649 (636h9m20.097167684s) [upload=557425]: Waiting for error group managing segments... Jan 1 03:38:20.500808796 (636h9m20.371891841s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.511146620 (636h9m20.382229665s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.512963139 (636h9m20.384046184s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.513585224 (636h9m20.384668269s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.517155291 (636h9m20.388238336s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.528961282 (636h9m20.400044317s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.530863914 (636h9m20.401946949s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.538983445 (636h9m20.41006649s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.595792125 (636h9m20.46687516s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.612204215 (636h9m20.48328725s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.767931659 (636h9m20.639014694s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.867883629 (636h9m20.738966664s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871885079 (636h9m20.742968114s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871902772 (636h9m20.742985807s) [seg_pos={0 0}, upload=557425]: Segment reached optimal threshold of 13 pieces. Jan 1 03:38:20.872093115 (636h9m20.74317618s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872105989 (636h9m20.743189024s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872304367 (636h9m20.743387402s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872325717 (636h9m20.743408772s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872342800 (636h9m20.743425946s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection Jan 1 03:38:20.872355464 (636h9m20.743438509s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872439024 (636h9m20.743522069s) [seg_pos={0 0}, upload=557425]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:38:20.872567558 (636h9m20.743650603s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:38:20.872573520 (636h9m20.743656555s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:38:21.172543628 (636h9m21.043626673s) [upload=557425]: Done uploading segments... Jan 1 03:38:21.172563316 (636h9m21.043646361s) [upload=557425]: Upload finished. err: <nil> Duration: 6.012516174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X....X........................................X.] duration: 35.947413164s trace: https://jaeger.dev.storj.io/trace/1ebdd84a47b0fea8 FAIL (2026-01-01 03:40:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555732730 (636h10m14.426815775s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555861985 (636h10m14.42694503s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.556167417 (636h10m14.427250462s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557132745 (636h10m14.42821578s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557290145 (636h10m14.42837318s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.558546016 (636h10m14.429629051s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.559810844 (636h10m14.430893889s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.561308566 (636h10m14.432391601s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.562571591 (636h10m14.433654636s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.563473288 (636h10m14.434556333s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.564781790 (636h10m14.435864825s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.565835766 (636h10m14.436918801s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567015342 (636h10m14.438098387s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567628530 (636h10m14.438711565s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:16.014065836 (636h10m15.885148871s) [upload=557441]: Next returned nil segment Jan 1 03:39:16.014083580 (636h10m15.885166615s) [upload=557441]: Waiting for error group managing segments... Jan 1 03:39:16.455400629 (636h10m16.326483664s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.464872484 (636h10m16.335955529s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.616875776 (636h10m16.487958821s) [noise=true , address=77.90.31.19:4876 , piece_id=UVRE5SNVM7SVUWD37QXCFDL7FWRQIMGFBYKLUULJKQJ4SW6224PA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.759240917 (636h10m16.630323952s) [noise=true , address=77.90.31.21:4907 , piece_id=XELNGQJ3EHMREVTXIFRR4FLXDD5U6CMAQ53GRSJ4WOT76QOQCBGQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.826797803 (636h10m16.697880848s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.039826124 (636h10m16.910909169s) [noise=true , address=145.79.227.49:4357 , piece_id=NRGFOQVKNCENLBXQDAU63ZQTJXYDJZ5YMEMZUHNBAWCERZCDI5EQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.319810887 (636h10m17.190893922s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.335275864 (636h10m17.206358899s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.360870549 (636h10m17.231953594s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.412439583 (636h10m17.283522628s) [noise=true , address=145.79.227.35:4513 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.469220361 (636h10m17.340303396s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.485589420 (636h10m17.356672455s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.598713200 (636h10m17.469796245s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911962313 (636h10m17.783045348s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911978394 (636h10m17.783061429s) [seg_pos={0 0}, upload=557441]: Segment reached optimal threshold of 14 pieces. Jan 1 03:39:17.912112600 (636h10m17.783195635s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912118220 (636h10m17.783201255s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912348508 (636h10m17.783431543s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912354901 (636h10m17.783437936s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912458238 (636h10m17.783541283s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection Jan 1 03:39:17.912463217 (636h10m17.783546252s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912551856 (636h10m17.783634891s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE): protocol: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection Jan 1 03:39:17.912557115 (636h10m17.783640151s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912582464 (636h10m17.783665499s) [seg_pos={0 0}, upload=557441]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:39:17.912645484 (636h10m17.783728519s) [upload=557441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:39:17.912649532 (636h10m17.783732567s) [upload=557441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:39:18.208729342 (636h10m18.079812378s) [upload=557441]: Done uploading segments... Jan 1 03:39:18.208744913 (636h10m18.079827958s) [upload=557441]: Upload finished. err: <nil> Duration: 35.947426428s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X......X............................X........X.] duration: 3.626487919s trace: https://jaeger.dev.storj.io/trace/8b9520ee0ddc710 FAIL (2026-01-01 03:40:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.626505764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....X........................................X] duration: 1m0.003113515s trace: https://jaeger.dev.storj.io/trace/1ebdd84a47b0fea8 FAIL (2026-01-01 03:40:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555732730 (636h10m14.426815775s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555861985 (636h10m14.42694503s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.556167417 (636h10m14.427250462s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557132745 (636h10m14.42821578s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557290145 (636h10m14.42837318s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.558546016 (636h10m14.429629051s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.559810844 (636h10m14.430893889s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.561308566 (636h10m14.432391601s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.562571591 (636h10m14.433654636s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.563473288 (636h10m14.434556333s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.564781790 (636h10m14.435864825s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.565835766 (636h10m14.436918801s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567015342 (636h10m14.438098387s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567628530 (636h10m14.438711565s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:16.014065836 (636h10m15.885148871s) [upload=557441]: Next returned nil segment Jan 1 03:39:16.014083580 (636h10m15.885166615s) [upload=557441]: Waiting for error group managing segments... Jan 1 03:39:16.455400629 (636h10m16.326483664s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.464872484 (636h10m16.335955529s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.616875776 (636h10m16.487958821s) [noise=true , address=77.90.31.19:4876 , piece_id=UVRE5SNVM7SVUWD37QXCFDL7FWRQIMGFBYKLUULJKQJ4SW6224PA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.759240917 (636h10m16.630323952s) [noise=true , address=77.90.31.21:4907 , piece_id=XELNGQJ3EHMREVTXIFRR4FLXDD5U6CMAQ53GRSJ4WOT76QOQCBGQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.826797803 (636h10m16.697880848s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.039826124 (636h10m16.910909169s) [noise=true , address=145.79.227.49:4357 , piece_id=NRGFOQVKNCENLBXQDAU63ZQTJXYDJZ5YMEMZUHNBAWCERZCDI5EQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.319810887 (636h10m17.190893922s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.335275864 (636h10m17.206358899s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.360870549 (636h10m17.231953594s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.412439583 (636h10m17.283522628s) [noise=true , address=145.79.227.35:4513 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.469220361 (636h10m17.340303396s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.485589420 (636h10m17.356672455s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.598713200 (636h10m17.469796245s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911962313 (636h10m17.783045348s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911978394 (636h10m17.783061429s) [seg_pos={0 0}, upload=557441]: Segment reached optimal threshold of 14 pieces. Jan 1 03:39:17.912112600 (636h10m17.783195635s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912118220 (636h10m17.783201255s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912348508 (636h10m17.783431543s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912354901 (636h10m17.783437936s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912458238 (636h10m17.783541283s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection Jan 1 03:39:17.912463217 (636h10m17.783546252s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912551856 (636h10m17.783634891s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE): protocol: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection Jan 1 03:39:17.912557115 (636h10m17.783640151s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912582464 (636h10m17.783665499s) [seg_pos={0 0}, upload=557441]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:39:17.912645484 (636h10m17.783728519s) [upload=557441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:39:17.912649532 (636h10m17.783732567s) [upload=557441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:39:18.208729342 (636h10m18.079812378s) [upload=557441]: Done uploading segments... Jan 1 03:39:18.208744913 (636h10m18.079827958s) [upload=557441]: Upload finished. err: <nil> Duration: 1m0.003136097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX..] duration: 6.072242599s trace: https://jaeger.dev.storj.io/trace/3a1918a7502a0c40 FAIL (2026-01-01 03:39:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:38:18.274492318 (636h9m18.145575353s) [upload=557425]: Starting upload Jan 1 03:38:18.274567070 (636h9m18.145650105s) [upload=557425]: Done starting upload Jan 1 03:38:18.274688662 (636h9m18.145771707s) [upload=557425]: Uploading segments... Jan 1 03:38:18.274806416 (636h9m18.145889461s) [upload=557425]: Got next segment. Inline: false Jan 1 03:38:18.274842244 (636h9m18.145925289s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:38:18.274847464 (636h9m18.145930519s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:38:18.522138205 (636h9m18.39322127s) [seg_pos={0 0}, upload=557425]: Begin upload segment... Jan 1 03:38:18.522195464 (636h9m18.393278509s) [seg_pos={0 0}, upload=557425]: Done begin upload segment. Jan 1 03:38:18.522222756 (636h9m18.393305791s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522240590 (636h9m18.393323655s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522841954 (636h9m18.393924999s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523203123 (636h9m18.394286168s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523238200 (636h9m18.394321245s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523581053 (636h9m18.394664088s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523675452 (636h9m18.394758487s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523829435 (636h9m18.39491247s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523980252 (636h9m18.395063287s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524079642 (636h9m18.395162677s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524396465 (636h9m18.3954795s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524481868 (636h9m18.395564943s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524653404 (636h9m18.395736439s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524906376 (636h9m18.395989411s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524912417 (636h9m18.395995452s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.525774960 (636h9m18.396857995s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:20.226070182 (636h9m20.097153217s) [upload=557425]: Next returned nil segment Jan 1 03:38:20.226084649 (636h9m20.097167684s) [upload=557425]: Waiting for error group managing segments... Jan 1 03:38:20.500808796 (636h9m20.371891841s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.511146620 (636h9m20.382229665s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.512963139 (636h9m20.384046184s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.513585224 (636h9m20.384668269s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.517155291 (636h9m20.388238336s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.528961282 (636h9m20.400044317s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.530863914 (636h9m20.401946949s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.538983445 (636h9m20.41006649s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.595792125 (636h9m20.46687516s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.612204215 (636h9m20.48328725s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.767931659 (636h9m20.639014694s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.867883629 (636h9m20.738966664s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871885079 (636h9m20.742968114s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871902772 (636h9m20.742985807s) [seg_pos={0 0}, upload=557425]: Segment reached optimal threshold of 13 pieces. Jan 1 03:38:20.872093115 (636h9m20.74317618s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872105989 (636h9m20.743189024s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872304367 (636h9m20.743387402s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872325717 (636h9m20.743408772s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872342800 (636h9m20.743425946s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection Jan 1 03:38:20.872355464 (636h9m20.743438509s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872439024 (636h9m20.743522069s) [seg_pos={0 0}, upload=557425]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:38:20.872567558 (636h9m20.743650603s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:38:20.872573520 (636h9m20.743656555s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:38:21.172543628 (636h9m21.043626673s) [upload=557425]: Done uploading segments... Jan 1 03:38:21.172563316 (636h9m21.043646361s) [upload=557425]: Upload finished. err: <nil> Duration: 6.07227494s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X......X............................X........X] duration: 5.001156187s trace: https://jaeger.dev.storj.io/trace/8b9520ee0ddc710 FAIL (2026-01-01 03:39:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001180133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX.] duration: 5.761767777s trace: https://jaeger.dev.storj.io/trace/3a1918a7502a0c40 FAIL (2026-01-01 03:39:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:38:18.274492318 (636h9m18.145575353s) [upload=557425]: Starting upload Jan 1 03:38:18.274567070 (636h9m18.145650105s) [upload=557425]: Done starting upload Jan 1 03:38:18.274688662 (636h9m18.145771707s) [upload=557425]: Uploading segments... Jan 1 03:38:18.274806416 (636h9m18.145889461s) [upload=557425]: Got next segment. Inline: false Jan 1 03:38:18.274842244 (636h9m18.145925289s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:38:18.274847464 (636h9m18.145930519s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:38:18.522138205 (636h9m18.39322127s) [seg_pos={0 0}, upload=557425]: Begin upload segment... Jan 1 03:38:18.522195464 (636h9m18.393278509s) [seg_pos={0 0}, upload=557425]: Done begin upload segment. Jan 1 03:38:18.522222756 (636h9m18.393305791s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522240590 (636h9m18.393323655s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522841954 (636h9m18.393924999s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523203123 (636h9m18.394286168s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523238200 (636h9m18.394321245s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523581053 (636h9m18.394664088s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523675452 (636h9m18.394758487s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523829435 (636h9m18.39491247s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523980252 (636h9m18.395063287s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524079642 (636h9m18.395162677s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524396465 (636h9m18.3954795s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524481868 (636h9m18.395564943s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524653404 (636h9m18.395736439s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524906376 (636h9m18.395989411s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524912417 (636h9m18.395995452s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.525774960 (636h9m18.396857995s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:20.226070182 (636h9m20.097153217s) [upload=557425]: Next returned nil segment Jan 1 03:38:20.226084649 (636h9m20.097167684s) [upload=557425]: Waiting for error group managing segments... Jan 1 03:38:20.500808796 (636h9m20.371891841s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.511146620 (636h9m20.382229665s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.512963139 (636h9m20.384046184s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.513585224 (636h9m20.384668269s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.517155291 (636h9m20.388238336s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.528961282 (636h9m20.400044317s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.530863914 (636h9m20.401946949s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.538983445 (636h9m20.41006649s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.595792125 (636h9m20.46687516s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.612204215 (636h9m20.48328725s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.767931659 (636h9m20.639014694s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.867883629 (636h9m20.738966664s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871885079 (636h9m20.742968114s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871902772 (636h9m20.742985807s) [seg_pos={0 0}, upload=557425]: Segment reached optimal threshold of 13 pieces. Jan 1 03:38:20.872093115 (636h9m20.74317618s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872105989 (636h9m20.743189024s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872304367 (636h9m20.743387402s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872325717 (636h9m20.743408772s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872342800 (636h9m20.743425946s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection Jan 1 03:38:20.872355464 (636h9m20.743438509s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872439024 (636h9m20.743522069s) [seg_pos={0 0}, upload=557425]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:38:20.872567558 (636h9m20.743650603s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:38:20.872573520 (636h9m20.743656555s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:38:21.172543628 (636h9m21.043626673s) [upload=557425]: Done uploading segments... Jan 1 03:38:21.172563316 (636h9m21.043646361s) [upload=557425]: Upload finished. err: <nil> Duration: 5.761799509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 29.008852313s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 29.008887641s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X......X............................X........] duration: 3.597429598s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:38:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.597461187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [....X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX] duration: 30.004150741s trace: https://jaeger.dev.storj.io/trace/3a1918a7502a0c40 FAIL (2026-01-01 03:38:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:38:18.274492318 (636h9m18.145575353s) [upload=557425]: Starting upload Jan 1 03:38:18.274567070 (636h9m18.145650105s) [upload=557425]: Done starting upload Jan 1 03:38:18.274688662 (636h9m18.145771707s) [upload=557425]: Uploading segments... Jan 1 03:38:18.274806416 (636h9m18.145889461s) [upload=557425]: Got next segment. Inline: false Jan 1 03:38:18.274842244 (636h9m18.145925289s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:38:18.274847464 (636h9m18.145930519s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:38:18.522138205 (636h9m18.39322127s) [seg_pos={0 0}, upload=557425]: Begin upload segment... Jan 1 03:38:18.522195464 (636h9m18.393278509s) [seg_pos={0 0}, upload=557425]: Done begin upload segment. Jan 1 03:38:18.522222756 (636h9m18.393305791s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522240590 (636h9m18.393323655s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522841954 (636h9m18.393924999s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523203123 (636h9m18.394286168s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523238200 (636h9m18.394321245s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523581053 (636h9m18.394664088s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523675452 (636h9m18.394758487s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523829435 (636h9m18.39491247s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523980252 (636h9m18.395063287s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524079642 (636h9m18.395162677s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524396465 (636h9m18.3954795s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524481868 (636h9m18.395564943s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524653404 (636h9m18.395736439s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524906376 (636h9m18.395989411s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524912417 (636h9m18.395995452s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.525774960 (636h9m18.396857995s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:20.226070182 (636h9m20.097153217s) [upload=557425]: Next returned nil segment Jan 1 03:38:20.226084649 (636h9m20.097167684s) [upload=557425]: Waiting for error group managing segments... Jan 1 03:38:20.500808796 (636h9m20.371891841s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.511146620 (636h9m20.382229665s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.512963139 (636h9m20.384046184s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.513585224 (636h9m20.384668269s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.517155291 (636h9m20.388238336s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.528961282 (636h9m20.400044317s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.530863914 (636h9m20.401946949s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.538983445 (636h9m20.41006649s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.595792125 (636h9m20.46687516s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.612204215 (636h9m20.48328725s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.767931659 (636h9m20.639014694s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.867883629 (636h9m20.738966664s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871885079 (636h9m20.742968114s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871902772 (636h9m20.742985807s) [seg_pos={0 0}, upload=557425]: Segment reached optimal threshold of 13 pieces. Jan 1 03:38:20.872093115 (636h9m20.74317618s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872105989 (636h9m20.743189024s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872304367 (636h9m20.743387402s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872325717 (636h9m20.743408772s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872342800 (636h9m20.743425946s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection Jan 1 03:38:20.872355464 (636h9m20.743438509s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872439024 (636h9m20.743522069s) [seg_pos={0 0}, upload=557425]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:38:20.872567558 (636h9m20.743650603s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:38:20.872573520 (636h9m20.743656555s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:38:21.172543628 (636h9m21.043626673s) [upload=557425]: Done uploading segments... Jan 1 03:38:21.172563316 (636h9m21.043646361s) [upload=557425]: Upload finished. err: <nil> Duration: 30.004172933s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X......X............................X.......] duration: 3.520895048s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:38:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520918812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....X] duration: 30.003116979s trace: https://jaeger.dev.storj.io/trace/549f4351e4519740 FAIL (2026-01-01 03:38:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (636h8m40.904428876s) [upload=557417]: Uploading segments... Jan 1 03:37:41.033431844 (636h8m40.904514889s) [upload=557417]: Got next segment. Inline: false Jan 1 03:37:41.033484755 (636h8m40.904567801s) [upload=557417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:37:41.033490837 (636h8m40.904573882s) [upload=557417]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:37:41.280960819 (636h8m41.152043854s) [seg_pos={0 0}, upload=557417]: Begin upload segment... Jan 1 03:37:41.281134160 (636h8m41.152217205s) [seg_pos={0 0}, upload=557417]: Done begin upload segment. Jan 1 03:37:41.281167002 (636h8m41.152250037s) [noise=true , address=145.79.227.8:4331 , piece_id=JJVBSDXEGXOQ7N3GLYGU4BPRFS2FWLD7DHHPZ5EXNIUF2ATYSJKQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.281258205 (636h8m41.15234125s) [noise=true , address=77.90.31.18:4805 , piece_id=YHNGUE42NSLAIOMYWQ4SJ2C7O4BUKUAEXTYEIJZYCKKZCBMOSFQQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.281703664 (636h8m41.152786699s) [noise=true , address=145.79.227.41:4087 , piece_id=7SV7CLVJGNZQJNQJPXN2NBDPOQ24QYN654PX6MFTJ62IVGI27BOQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.282189188 (636h8m41.153272223s) [noise=true , address=145.79.227.31:4797 , piece_id=QEXJ5KUPNJZ5PAISMBNMPCHQL2ZQL5IEMRESHOIBOYMWG22NGVLQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.282690502 (636h8m41.153773547s) [noise=true , address=45.91.116.4:4940 , piece_id=DVLVAHFQIIIJLUV5AC6X6CNPULNVLFQBTJ5BNUP2SMJV5Y2MSZJA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.282817735 (636h8m41.15390077s) [noise=true , address=145.79.227.39:4013 , piece_id=LYFILPOG4UVX4ISNS73MXSXIQDMGE4YB7W7YNM5NV3RFRMYQVLAA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283101575 (636h8m41.15418461s) [noise=true , address=77.90.31.16:4735 , piece_id=AZGPEWUICPQSCS7WFXXKSW3ALI5IRKCMJDUSAWRK6LPWUESTIU4A, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283205664 (636h8m41.154288699s) [noise=true , address=145.79.227.2:4113 , piece_id=WCOX5PFXBPSNAUDO5OSOBNSPIV3UX5GUAJ4ORRSPQYP5RCAFJEZQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283479896 (636h8m41.154562941s) [noise=true , address=77.90.31.11:4554 , piece_id=744L33MQ3O5M222DKOK4Z6ZHRSRLH72FXFUACXVWTUKWQ6ZZGQIA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283551833 (636h8m41.154634868s) [noise=true , address=145.79.227.6:4258 , piece_id=IG6IF6UJ6RLKKSTMJ6IISJNZHFZWHFX7GHXOXP44AAKZGV4GNSWQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283796208 (636h8m41.154879253s) [noise=true , address=45.91.116.5:4297 , piece_id=35SRYKGA36EQK6YQWINWYOAZPHZTHRPXWLUPR36SR2NNHW6CBITQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283884186 (636h8m41.154967221s) [noise=true , address=145.79.227.45:4230 , piece_id=2C3CMTNXCR5G2ZHV4REX4EGVEF4RE2OD5E2SEAPOZKBIG5OTXFLQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.284141165 (636h8m41.1552242s) [noise=true , address=145.79.227.27:4689 , piece_id=MXHNHYBRABN23H43MCGSPV66VZO377NO5F7RYCSXL7HSGIAKYRXA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.284223702 (636h8m41.155306737s) [noise=true , address=145.79.227.42:4122 , piece_id=P24QWODKVFQPUKYCITZ5FEUDLRKIWS7OIW46UR2BXKS4Z2YDHB4Q, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.284521690 (636h8m41.155604735s) [noise=true , address=77.90.31.13:4627 , piece_id=75QZDXBRQOV7SDDC5F6HTPUVM6EDV5MMTNROBFZNCNIR7QH3GVRA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.284696733 (636h8m41.155779778s) [noise=true , address=45.91.116.10:4082 , piece_id=RALN6RUHCQJ5H674VQD5J4NEJWKP2QFWRQBPNBB5OEORWD4YWUTA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:42.984632610 (636h8m42.855715645s) [upload=557417]: Next returned nil segment Jan 1 03:37:42.984647118 (636h8m42.855730153s) [upload=557417]: Waiting for error group managing segments... Jan 1 03:37:43.256857678 (636h8m43.127940723s) [noise=true , address=145.79.227.45:4230 , piece_id=2C3CMTNXCR5G2ZHV4REX4EGVEF4RE2OD5E2SEAPOZKBIG5OTXFLQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.266329344 (636h8m43.137412379s) [noise=true , address=145.79.227.41:4087 , piece_id=7SV7CLVJGNZQJNQJPXN2NBDPOQ24QYN654PX6MFTJ62IVGI27BOQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.269930430 (636h8m43.141013476s) [noise=true , address=145.79.227.2:4113 , piece_id=WCOX5PFXBPSNAUDO5OSOBNSPIV3UX5GUAJ4ORRSPQYP5RCAFJEZQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.275533440 (636h8m43.146616475s) [noise=true , address=145.79.227.27:4689 , piece_id=MXHNHYBRABN23H43MCGSPV66VZO377NO5F7RYCSXL7HSGIAKYRXA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.282519421 (636h8m43.153602457s) [noise=true , address=145.79.227.6:4258 , piece_id=IG6IF6UJ6RLKKSTMJ6IISJNZHFZWHFX7GHXOXP44AAKZGV4GNSWQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.286136619 (636h8m43.157219654s) [noise=true , address=145.79.227.8:4331 , piece_id=JJVBSDXEGXOQ7N3GLYGU4BPRFS2FWLD7DHHPZ5EXNIUF2ATYSJKQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.289643076 (636h8m43.160726112s) [noise=true , address=145.79.227.31:4797 , piece_id=QEXJ5KUPNJZ5PAISMBNMPCHQL2ZQL5IEMRESHOIBOYMWG22NGVLQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.355713682 (636h8m43.226796727s) [noise=true , address=145.79.227.39:4013 , piece_id=LYFILPOG4UVX4ISNS73MXSXIQDMGE4YB7W7YNM5NV3RFRMYQVLAA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.360679467 (636h8m43.231762512s) [noise=true , address=145.79.227.42:4122 , piece_id=P24QWODKVFQPUKYCITZ5FEUDLRKIWS7OIW46UR2BXKS4Z2YDHB4Q, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.547870661 (636h8m43.418953696s) [noise=true , address=77.90.31.11:4554 , piece_id=744L33MQ3O5M222DKOK4Z6ZHRSRLH72FXFUACXVWTUKWQ6ZZGQIA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.584330184 (636h8m43.455413219s) [noise=true , address=77.90.31.18:4805 , piece_id=YHNGUE42NSLAIOMYWQ4SJ2C7O4BUKUAEXTYEIJZYCKKZCBMOSFQQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.615042174 (636h8m43.486125219s) [noise=true , address=77.90.31.16:4735 , piece_id=AZGPEWUICPQSCS7WFXXKSW3ALI5IRKCMJDUSAWRK6LPWUESTIU4A, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.687030308 (636h8m43.558113343s) [noise=true , address=45.91.116.5:4297 , piece_id=35SRYKGA36EQK6YQWINWYOAZPHZTHRPXWLUPR36SR2NNHW6CBITQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.687054985 (636h8m43.55813802s) [seg_pos={0 0}, upload=557417]: Segment reached optimal threshold of 13 pieces. Jan 1 03:37:43.687230439 (636h8m43.558313474s) [noise=true , address=77.90.31.13:4627 , piece_id=75QZDXBRQOV7SDDC5F6HTPUVM6EDV5MMTNROBFZNCNIR7QH3GVRA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:55016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:55016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55016->77.90.31.13:4627: use of closed network connection Jan 1 03:37:43.687236180 (636h8m43.558319225s) [noise=true , address=77.90.31.13:4627 , piece_id=75QZDXBRQOV7SDDC5F6HTPUVM6EDV5MMTNROBFZNCNIR7QH3GVRA, seg_pos={0 0}, upload=557417]: Long tail cancellation due to optimal threshold Jan 1 03:37:43.687355567 (636h8m43.558438602s) [noise=true , address=45.91.116.4:4940 , piece_id=DVLVAHFQIIIJLUV5AC6X6CNPULNVLFQBTJ5BNUP2SMJV5Y2MSZJA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:56256->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.4:4940: use of closed network connection Jan 1 03:37:43.687362780 (636h8m43.558445815s) [noise=true , address=45.91.116.4:4940 , piece_id=DVLVAHFQIIIJLUV5AC6X6CNPULNVLFQBTJ5BNUP2SMJV5Y2MSZJA, seg_pos={0 0}, upload=557417]: Long tail cancellation due to optimal threshold Jan 1 03:37:43.687481136 (636h8m43.558564181s) [noise=true , address=45.91.116.10:4082 , piece_id=RALN6RUHCQJ5H674VQD5J4NEJWKP2QFWRQBPNBB5OEORWD4YWUTA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 1 03:37:43.687487148 (636h8m43.558570193s) [noise=true , address=45.91.116.10:4082 , piece_id=RALN6RUHCQJ5H674VQD5J4NEJWKP2QFWRQBPNBB5OEORWD4YWUTA, seg_pos={0 0}, upload=557417]: Long tail cancellation due to optimal threshold Jan 1 03:37:43.687502867 (636h8m43.558585912s) [seg_pos={0 0}, upload=557417]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:37:43.687568442 (636h8m43.558651477s) [upload=557417]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:37:43.687573622 (636h8m43.558656667s) [upload=557417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:37:43.980870394 (636h8m43.851953439s) [upload=557417]: Done uploading segments... Jan 1 03:37:43.980893307 (636h8m43.851976352s) [upload=557417]: Upload finished. err: <nil> Duration: 30.00314372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X....X.......................................] duration: 42.122057693s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:37:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 42.122092498s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X......X............................X......] duration: 3.486978585s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:37:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487005527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X....X......................................] duration: 45.405673778s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 45.405704056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....] duration: 7.244847032s trace: https://jaeger.dev.storj.io/trace/511c0acf1687258e FAIL (2026-01-01 03:37:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA) context deadline exceeded ======LOGS====== Jan 1 03:33:56.314733719 (636h4m56.185816764s) [upload=557361]: Starting upload Jan 1 03:33:56.314856132 (636h4m56.185939177s) [upload=557361]: Done starting upload Jan 1 03:33:56.314936174 (636h4m56.186019209s) [upload=557361]: Uploading segments... Jan 1 03:33:56.315127339 (636h4m56.186210374s) [upload=557361]: Got next segment. Inline: false Jan 1 03:33:56.315162365 (636h4m56.186245411s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:33:56.315167675 (636h4m56.186250721s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:33:56.585484360 (636h4m56.456567405s) [seg_pos={0 0}, upload=557361]: Begin upload segment... Jan 1 03:33:56.585563410 (636h4m56.456646455s) [seg_pos={0 0}, upload=557361]: Done begin upload segment. Jan 1 03:33:56.585600531 (636h4m56.456683576s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.585748682 (636h4m56.456831728s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.586623668 (636h4m56.457706704s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587286000 (636h4m56.458369035s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587953671 (636h4m56.459036716s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.588896376 (636h4m56.459979421s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.590458651 (636h4m56.461541716s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.592544222 (636h4m56.463627297s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.593386686 (636h4m56.464469731s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594256092 (636h4m56.465339127s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594836167 (636h4m56.465919202s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.595705282 (636h4m56.466788327s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596215913 (636h4m56.467298948s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596517317 (636h4m56.467600363s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.598204309 (636h4m56.469287344s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.600691746 (636h4m56.471774791s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:58.281710480 (636h4m58.152793525s) [upload=557361]: Next returned nil segment Jan 1 03:33:58.281724658 (636h4m58.152807693s) [upload=557361]: Waiting for error group managing segments... Jan 1 03:33:58.566383030 (636h4m58.437466075s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.567784098 (636h4m58.438867173s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.583853285 (636h4m58.45493633s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.591537447 (636h4m58.462620492s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.592304357 (636h4m58.463387392s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.599833965 (636h4m58.47091701s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.601388594 (636h4m58.472471639s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.630591339 (636h4m58.501674374s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.650577436 (636h4m58.521660481s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.916297107 (636h4m58.787380152s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.920798298 (636h4m58.791881333s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.921432575 (636h4m58.792515621s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955733448 (636h4m58.826816493s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955763956 (636h4m58.826846991s) [seg_pos={0 0}, upload=557361]: Segment reached optimal threshold of 13 pieces. Jan 1 03:33:58.956096580 (636h4m58.827179625s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956105156 (636h4m58.827188201s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956367546 (636h4m58.827450591s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956391661 (636h4m58.827474706s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956461695 (636h4m58.82754476s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection Jan 1 03:33:58.956474369 (636h4m58.827557404s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956524344 (636h4m58.827607389s) [seg_pos={0 0}, upload=557361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:33:58.956658219 (636h4m58.827741274s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:33:58.956667076 (636h4m58.827750111s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:33:59.288606697 (636h4m59.159689732s) [upload=557361]: Done uploading segments... Jan 1 03:33:59.288632837 (636h4m59.159715882s) [upload=557361]: Upload finished. err: <nil> Duration: 7.244885054s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X......X............................X.....] duration: 3.542189467s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:36:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542220525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X....X.....................................] duration: 36.589320083s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 36.589349659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX...] duration: 5.991020187s trace: https://jaeger.dev.storj.io/trace/511c0acf1687258e FAIL (2026-01-01 03:36:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA) context deadline exceeded ======LOGS====== Jan 1 03:33:56.314733719 (636h4m56.185816764s) [upload=557361]: Starting upload Jan 1 03:33:56.314856132 (636h4m56.185939177s) [upload=557361]: Done starting upload Jan 1 03:33:56.314936174 (636h4m56.186019209s) [upload=557361]: Uploading segments... Jan 1 03:33:56.315127339 (636h4m56.186210374s) [upload=557361]: Got next segment. Inline: false Jan 1 03:33:56.315162365 (636h4m56.186245411s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:33:56.315167675 (636h4m56.186250721s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:33:56.585484360 (636h4m56.456567405s) [seg_pos={0 0}, upload=557361]: Begin upload segment... Jan 1 03:33:56.585563410 (636h4m56.456646455s) [seg_pos={0 0}, upload=557361]: Done begin upload segment. Jan 1 03:33:56.585600531 (636h4m56.456683576s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.585748682 (636h4m56.456831728s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.586623668 (636h4m56.457706704s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587286000 (636h4m56.458369035s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587953671 (636h4m56.459036716s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.588896376 (636h4m56.459979421s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.590458651 (636h4m56.461541716s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.592544222 (636h4m56.463627297s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.593386686 (636h4m56.464469731s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594256092 (636h4m56.465339127s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594836167 (636h4m56.465919202s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.595705282 (636h4m56.466788327s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596215913 (636h4m56.467298948s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596517317 (636h4m56.467600363s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.598204309 (636h4m56.469287344s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.600691746 (636h4m56.471774791s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:58.281710480 (636h4m58.152793525s) [upload=557361]: Next returned nil segment Jan 1 03:33:58.281724658 (636h4m58.152807693s) [upload=557361]: Waiting for error group managing segments... Jan 1 03:33:58.566383030 (636h4m58.437466075s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.567784098 (636h4m58.438867173s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.583853285 (636h4m58.45493633s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.591537447 (636h4m58.462620492s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.592304357 (636h4m58.463387392s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.599833965 (636h4m58.47091701s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.601388594 (636h4m58.472471639s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.630591339 (636h4m58.501674374s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.650577436 (636h4m58.521660481s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.916297107 (636h4m58.787380152s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.920798298 (636h4m58.791881333s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.921432575 (636h4m58.792515621s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955733448 (636h4m58.826816493s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955763956 (636h4m58.826846991s) [seg_pos={0 0}, upload=557361]: Segment reached optimal threshold of 13 pieces. Jan 1 03:33:58.956096580 (636h4m58.827179625s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956105156 (636h4m58.827188201s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956367546 (636h4m58.827450591s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956391661 (636h4m58.827474706s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956461695 (636h4m58.82754476s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection Jan 1 03:33:58.956474369 (636h4m58.827557404s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956524344 (636h4m58.827607389s) [seg_pos={0 0}, upload=557361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:33:58.956658219 (636h4m58.827741274s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:33:58.956667076 (636h4m58.827750111s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:33:59.288606697 (636h4m59.159689732s) [upload=557361]: Done uploading segments... Jan 1 03:33:59.288632837 (636h4m59.159715882s) [upload=557361]: Upload finished. err: <nil> Duration: 5.991073608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X......X............................X....] duration: 3.674382762s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:35:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.674408841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X....X....................................] duration: 36.570470114s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:35:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 36.570485604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX..] duration: 8.047913459s trace: https://jaeger.dev.storj.io/trace/511c0acf1687258e FAIL (2026-01-01 03:35:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA) context deadline exceeded ======LOGS====== Jan 1 03:33:56.314733719 (636h4m56.185816764s) [upload=557361]: Starting upload Jan 1 03:33:56.314856132 (636h4m56.185939177s) [upload=557361]: Done starting upload Jan 1 03:33:56.314936174 (636h4m56.186019209s) [upload=557361]: Uploading segments... Jan 1 03:33:56.315127339 (636h4m56.186210374s) [upload=557361]: Got next segment. Inline: false Jan 1 03:33:56.315162365 (636h4m56.186245411s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:33:56.315167675 (636h4m56.186250721s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:33:56.585484360 (636h4m56.456567405s) [seg_pos={0 0}, upload=557361]: Begin upload segment... Jan 1 03:33:56.585563410 (636h4m56.456646455s) [seg_pos={0 0}, upload=557361]: Done begin upload segment. Jan 1 03:33:56.585600531 (636h4m56.456683576s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.585748682 (636h4m56.456831728s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.586623668 (636h4m56.457706704s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587286000 (636h4m56.458369035s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587953671 (636h4m56.459036716s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.588896376 (636h4m56.459979421s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.590458651 (636h4m56.461541716s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.592544222 (636h4m56.463627297s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.593386686 (636h4m56.464469731s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594256092 (636h4m56.465339127s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594836167 (636h4m56.465919202s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.595705282 (636h4m56.466788327s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596215913 (636h4m56.467298948s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596517317 (636h4m56.467600363s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.598204309 (636h4m56.469287344s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.600691746 (636h4m56.471774791s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:58.281710480 (636h4m58.152793525s) [upload=557361]: Next returned nil segment Jan 1 03:33:58.281724658 (636h4m58.152807693s) [upload=557361]: Waiting for error group managing segments... Jan 1 03:33:58.566383030 (636h4m58.437466075s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.567784098 (636h4m58.438867173s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.583853285 (636h4m58.45493633s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.591537447 (636h4m58.462620492s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.592304357 (636h4m58.463387392s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.599833965 (636h4m58.47091701s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.601388594 (636h4m58.472471639s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.630591339 (636h4m58.501674374s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.650577436 (636h4m58.521660481s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.916297107 (636h4m58.787380152s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.920798298 (636h4m58.791881333s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.921432575 (636h4m58.792515621s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955733448 (636h4m58.826816493s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955763956 (636h4m58.826846991s) [seg_pos={0 0}, upload=557361]: Segment reached optimal threshold of 13 pieces. Jan 1 03:33:58.956096580 (636h4m58.827179625s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956105156 (636h4m58.827188201s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956367546 (636h4m58.827450591s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956391661 (636h4m58.827474706s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956461695 (636h4m58.82754476s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection Jan 1 03:33:58.956474369 (636h4m58.827557404s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956524344 (636h4m58.827607389s) [seg_pos={0 0}, upload=557361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:33:58.956658219 (636h4m58.827741274s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:33:58.956667076 (636h4m58.827750111s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:33:59.288606697 (636h4m59.159689732s) [upload=557361]: Done uploading segments... Jan 1 03:33:59.288632837 (636h4m59.159715882s) [upload=557361]: Upload finished. err: <nil> Duration: 8.047943496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X....X...................................] duration: 50.843115285s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:35:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 50.843148538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X......X............................X...] duration: 4.008526539s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:35:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.008555823s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX.] duration: 6.163481535s trace: https://jaeger.dev.storj.io/trace/511c0acf1687258e FAIL (2026-01-01 03:34:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA) context deadline exceeded ======LOGS====== Jan 1 03:33:56.314733719 (636h4m56.185816764s) [upload=557361]: Starting upload Jan 1 03:33:56.314856132 (636h4m56.185939177s) [upload=557361]: Done starting upload Jan 1 03:33:56.314936174 (636h4m56.186019209s) [upload=557361]: Uploading segments... Jan 1 03:33:56.315127339 (636h4m56.186210374s) [upload=557361]: Got next segment. Inline: false Jan 1 03:33:56.315162365 (636h4m56.186245411s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:33:56.315167675 (636h4m56.186250721s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:33:56.585484360 (636h4m56.456567405s) [seg_pos={0 0}, upload=557361]: Begin upload segment... Jan 1 03:33:56.585563410 (636h4m56.456646455s) [seg_pos={0 0}, upload=557361]: Done begin upload segment. Jan 1 03:33:56.585600531 (636h4m56.456683576s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.585748682 (636h4m56.456831728s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.586623668 (636h4m56.457706704s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587286000 (636h4m56.458369035s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587953671 (636h4m56.459036716s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.588896376 (636h4m56.459979421s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.590458651 (636h4m56.461541716s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.592544222 (636h4m56.463627297s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.593386686 (636h4m56.464469731s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594256092 (636h4m56.465339127s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594836167 (636h4m56.465919202s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.595705282 (636h4m56.466788327s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596215913 (636h4m56.467298948s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596517317 (636h4m56.467600363s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.598204309 (636h4m56.469287344s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.600691746 (636h4m56.471774791s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:58.281710480 (636h4m58.152793525s) [upload=557361]: Next returned nil segment Jan 1 03:33:58.281724658 (636h4m58.152807693s) [upload=557361]: Waiting for error group managing segments... Jan 1 03:33:58.566383030 (636h4m58.437466075s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.567784098 (636h4m58.438867173s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.583853285 (636h4m58.45493633s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.591537447 (636h4m58.462620492s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.592304357 (636h4m58.463387392s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.599833965 (636h4m58.47091701s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.601388594 (636h4m58.472471639s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.630591339 (636h4m58.501674374s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.650577436 (636h4m58.521660481s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.916297107 (636h4m58.787380152s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.920798298 (636h4m58.791881333s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.921432575 (636h4m58.792515621s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955733448 (636h4m58.826816493s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955763956 (636h4m58.826846991s) [seg_pos={0 0}, upload=557361]: Segment reached optimal threshold of 13 pieces. Jan 1 03:33:58.956096580 (636h4m58.827179625s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956105156 (636h4m58.827188201s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956367546 (636h4m58.827450591s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956391661 (636h4m58.827474706s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956461695 (636h4m58.82754476s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection Jan 1 03:33:58.956474369 (636h4m58.827557404s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956524344 (636h4m58.827607389s) [seg_pos={0 0}, upload=557361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:33:58.956658219 (636h4m58.827741274s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:33:58.956667076 (636h4m58.827750111s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:33:59.288606697 (636h4m59.159689732s) [upload=557361]: Done uploading segments... Jan 1 03:33:59.288632837 (636h4m59.159715882s) [upload=557361]: Upload finished. err: <nil> Duration: 6.163510631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX] duration: 30.00353164s trace: https://jaeger.dev.storj.io/trace/511c0acf1687258e FAIL (2026-01-01 03:34:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA) context deadline exceeded ======LOGS====== Jan 1 03:33:56.314733719 (636h4m56.185816764s) [upload=557361]: Starting upload Jan 1 03:33:56.314856132 (636h4m56.185939177s) [upload=557361]: Done starting upload Jan 1 03:33:56.314936174 (636h4m56.186019209s) [upload=557361]: Uploading segments... Jan 1 03:33:56.315127339 (636h4m56.186210374s) [upload=557361]: Got next segment. Inline: false Jan 1 03:33:56.315162365 (636h4m56.186245411s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:33:56.315167675 (636h4m56.186250721s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:33:56.585484360 (636h4m56.456567405s) [seg_pos={0 0}, upload=557361]: Begin upload segment... Jan 1 03:33:56.585563410 (636h4m56.456646455s) [seg_pos={0 0}, upload=557361]: Done begin upload segment. Jan 1 03:33:56.585600531 (636h4m56.456683576s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.585748682 (636h4m56.456831728s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.586623668 (636h4m56.457706704s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587286000 (636h4m56.458369035s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587953671 (636h4m56.459036716s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.588896376 (636h4m56.459979421s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.590458651 (636h4m56.461541716s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.592544222 (636h4m56.463627297s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.593386686 (636h4m56.464469731s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594256092 (636h4m56.465339127s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594836167 (636h4m56.465919202s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.595705282 (636h4m56.466788327s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596215913 (636h4m56.467298948s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596517317 (636h4m56.467600363s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.598204309 (636h4m56.469287344s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.600691746 (636h4m56.471774791s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:58.281710480 (636h4m58.152793525s) [upload=557361]: Next returned nil segment Jan 1 03:33:58.281724658 (636h4m58.152807693s) [upload=557361]: Waiting for error group managing segments... Jan 1 03:33:58.566383030 (636h4m58.437466075s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.567784098 (636h4m58.438867173s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.583853285 (636h4m58.45493633s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.591537447 (636h4m58.462620492s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.592304357 (636h4m58.463387392s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.599833965 (636h4m58.47091701s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.601388594 (636h4m58.472471639s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.630591339 (636h4m58.501674374s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.650577436 (636h4m58.521660481s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.916297107 (636h4m58.787380152s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.920798298 (636h4m58.791881333s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.921432575 (636h4m58.792515621s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955733448 (636h4m58.826816493s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955763956 (636h4m58.826846991s) [seg_pos={0 0}, upload=557361]: Segment reached optimal threshold of 13 pieces. Jan 1 03:33:58.956096580 (636h4m58.827179625s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956105156 (636h4m58.827188201s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956367546 (636h4m58.827450591s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956391661 (636h4m58.827474706s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956461695 (636h4m58.82754476s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection Jan 1 03:33:58.956474369 (636h4m58.827557404s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956524344 (636h4m58.827607389s) [seg_pos={0 0}, upload=557361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:33:58.956658219 (636h4m58.827741274s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:33:58.956667076 (636h4m58.827750111s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:33:59.288606697 (636h4m59.159689732s) [upload=557361]: Done uploading segments... Jan 1 03:33:59.288632837 (636h4m59.159715882s) [upload=557361]: Upload finished. err: <nil> Duration: 30.003575052s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X......X............................X..] duration: 4.024954559s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:33:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.025094193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X....X..................................] duration: 34.105672381s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:33:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 34.10570254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXX] duration: 30.00467481s trace: https://jaeger.dev.storj.io/trace/580b40c83d5cd86b FAIL (2026-01-01 03:33:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: RNYWJ7V2FXFR2TSJQA74MJDOBA6N3ELXXUNT3SIQZBSE7SL72W2Q) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: MFSZBF6TEYAXDEPNRNBSN7W74S6KI2OYSHJOOMMXPZTMSLJHC4SQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 5OLGIAFDSEPAE3SOYELXSCQU3TXI4NYTDSBW6ADWQV76AU5NG5CQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: VGH4WC5WMQ54736XZ2PDKMHS27VQ7J6CBHZI3TV6NKHJP3G54CVA) context deadline exceeded ======LOGS====== Jan 1 03:32:55.709775520 (636h3m55.580858565s) [upload=557348]: Starting upload Jan 1 03:32:55.709860711 (636h3m55.580943757s) [upload=557348]: Done starting upload Jan 1 03:32:55.709946325 (636h3m55.58102937s) [upload=557348]: Uploading segments... Jan 1 03:32:55.710112120 (636h3m55.581195155s) [upload=557348]: Got next segment. Inline: false Jan 1 03:32:55.710148549 (636h3m55.581231584s) [upload=557348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:32:55.710153560 (636h3m55.581236595s) [upload=557348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:32:55.955853521 (636h3m55.826936566s) [seg_pos={0 0}, upload=557348]: Begin upload segment... Jan 1 03:32:55.955911430 (636h3m55.826994465s) [seg_pos={0 0}, upload=557348]: Done begin upload segment. Jan 1 03:32:55.955957579 (636h3m55.827040614s) [noise=true , address=145.79.227.18:4006 , piece_id=OV2GBZFSO34TJRBFVY3ZY6OEESSCEUES3YYKFIGZVE3TSDNZG4AA, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.955992275 (636h3m55.82707531s) [noise=true , address=77.90.31.21:4880 , piece_id=RNYWJ7V2FXFR2TSJQA74MJDOBA6N3ELXXUNT3SIQZBSE7SL72W2Q, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.956054934 (636h3m55.827137979s) [noise=true , address=145.79.227.52:4447 , piece_id=VROQ7OP7TQ6BU53NQNDPRVZ4DZPLOLRWUPRWBHRJDJH2YG7HNERA, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.956646029 (636h3m55.827729064s) [noise=true , address=145.79.227.37:4565 , piece_id=GRVTGGMC66ULVCMTFZTIZJUEW3ZD7PR6UYJTHM3YPIQK6X5UJSDQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.956777540 (636h3m55.827860585s) [noise=true , address=45.91.116.9:4264 , piece_id=VVK3P3Y4M5Y3V4FBCOUXB7R4EMYGV4Q7SCX6GBONJ7HHSAAL5CMQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.957448437 (636h3m55.828531472s) [noise=true , address=145.79.227.2:4115 , piece_id=4BS4DSC7HIGFQQF42772655RMCPBC3Y2MTD7MRTYELCR6JE765EA, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.958688408 (636h3m55.829771443s) [noise=true , address=145.79.227.10:4403 , piece_id=NG4P6GSE5X7AKAEQ6N6GM4ZAWDFJ7EWVNEFZZ4DOCOAVINQEMYBA, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.959635582 (636h3m55.830718617s) [noise=true , address=77.90.31.13:4627 , piece_id=MFSZBF6TEYAXDEPNRNBSN7W74S6KI2OYSHJOOMMXPZTMSLJHC4SQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.960200277 (636h3m55.831283322s) [noise=true , address=45.91.116.14:4190 , piece_id=VGH4WC5WMQ54736XZ2PDKMHS27VQ7J6CBHZI3TV6NKHJP3G54CVA, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.960643611 (636h3m55.831726656s) [noise=true , address=145.79.227.38:4977 , piece_id=BX7TGUH7FPQWR3IGWYF5FW76TEJ6ZD3P3RR5KZEC7363R2RQ5GIQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.961244365 (636h3m55.83232741s) [noise=true , address=77.90.31.17:4773 , piece_id=YMMIFWC3IKTPKPM3XVPOVTLOPFJGIY6TNSUJQPWIMBGVKN6OWT7Q, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.961570465 (636h3m55.832653511s) [noise=true , address=145.79.227.20:4438 , piece_id=YR7QBFD7ICGTZ6YUXQB37ZLSG2FG46G6K235UOQ4AY6GCKCYEM6Q, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.961887379 (636h3m55.832970414s) [noise=true , address=145.79.227.33:4870 , piece_id=R3HM5OE54LCBKOLXQB3YAGMIEHNS2736PGVZNOE32IF36G6FAX5A, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.962068344 (636h3m55.833151379s) [noise=true , address=45.91.116.12:4226 , piece_id=5OLGIAFDSEPAE3SOYELXSCQU3TXI4NYTDSBW6ADWQV76AU5NG5CQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.962943641 (636h3m55.834026676s) [noise=true , address=77.90.31.10:4518 , piece_id=G52XWUBOIH6MBY4QMNNXV43TNU243QLFNOVXYHEE5WSCQGM5EU4Q, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.963182885 (636h3m55.834265921s) [noise=true , address=145.79.227.45:4230 , piece_id=7E4AHPDT2TPTQX5NA25JTWMXOWU4FN3ZJDCTB5G5O5OS4LB5K2RQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:57.652445537 (636h3m57.523528582s) [upload=557348]: Next returned nil segment Jan 1 03:32:57.652484081 (636h3m57.523567116s) [upload=557348]: Waiting for error group managing segments... Jan 1 03:32:57.936564262 (636h3m57.807647297s) [noise=true , address=145.79.227.45:4230 , piece_id=7E4AHPDT2TPTQX5NA25JTWMXOWU4FN3ZJDCTB5G5O5OS4LB5K2RQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.944531322 (636h3m57.815614368s) [noise=true , address=145.79.227.10:4403 , piece_id=NG4P6GSE5X7AKAEQ6N6GM4ZAWDFJ7EWVNEFZZ4DOCOAVINQEMYBA, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.952735855 (636h3m57.8238189s) [noise=true , address=145.79.227.37:4565 , piece_id=GRVTGGMC66ULVCMTFZTIZJUEW3ZD7PR6UYJTHM3YPIQK6X5UJSDQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.957181489 (636h3m57.828264524s) [noise=true , address=145.79.227.20:4438 , piece_id=YR7QBFD7ICGTZ6YUXQB37ZLSG2FG46G6K235UOQ4AY6GCKCYEM6Q, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.971171138 (636h3m57.842254173s) [noise=true , address=145.79.227.52:4447 , piece_id=VROQ7OP7TQ6BU53NQNDPRVZ4DZPLOLRWUPRWBHRJDJH2YG7HNERA, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.973415100 (636h3m57.844498145s) [noise=true , address=145.79.227.2:4115 , piece_id=4BS4DSC7HIGFQQF42772655RMCPBC3Y2MTD7MRTYELCR6JE765EA, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.980749576 (636h3m57.851832611s) [noise=true , address=145.79.227.38:4977 , piece_id=BX7TGUH7FPQWR3IGWYF5FW76TEJ6ZD3P3RR5KZEC7363R2RQ5GIQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:58.029969767 (636h3m57.901052812s) [noise=true , address=145.79.227.18:4006 , piece_id=OV2GBZFSO34TJRBFVY3ZY6OEESSCEUES3YYKFIGZVE3TSDNZG4AA, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:58.154161673 (636h3m58.025244728s) [noise=true , address=77.90.31.13:4627 , piece_id=MFSZBF6TEYAXDEPNRNBSN7W74S6KI2OYSHJOOMMXPZTMSLJHC4SQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:58.232702424 (636h3m58.103785459s) [noise=true , address=77.90.31.10:4518 , piece_id=G52XWUBOIH6MBY4QMNNXV43TNU243QLFNOVXYHEE5WSCQGM5EU4Q, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:58.240396674 (636h3m58.111479719s) [noise=true , address=77.90.31.21:4880 , piece_id=RNYWJ7V2FXFR2TSJQA74MJDOBA6N3ELXXUNT3SIQZBSE7SL72W2Q, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:59.027232650 (636h3m58.898315695s) [noise=true , address=45.91.116.14:4190 , piece_id=VGH4WC5WMQ54736XZ2PDKMHS27VQ7J6CBHZI3TV6NKHJP3G54CVA, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:59.297714539 (636h3m59.168797585s) [noise=true , address=45.91.116.12:4226 , piece_id=5OLGIAFDSEPAE3SOYELXSCQU3TXI4NYTDSBW6ADWQV76AU5NG5CQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:59.297741691 (636h3m59.168824736s) [seg_pos={0 0}, upload=557348]: Segment reached optimal threshold of 13 pieces. Jan 1 03:32:59.298014130 (636h3m59.169097185s) [noise=true , address=145.79.227.33:4870 , piece_id=R3HM5OE54LCBKOLXQB3YAGMIEHNS2736PGVZNOE32IF36G6FAX5A, seg_pos={0 0}, upload=557348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:59.298022736 (636h3m59.169105772s) [noise=true , address=145.79.227.33:4870 , piece_id=R3HM5OE54LCBKOLXQB3YAGMIEHNS2736PGVZNOE32IF36G6FAX5A, seg_pos={0 0}, upload=557348]: Long tail cancellation due to optimal threshold Jan 1 03:32:59.298335512 (636h3m59.169418547s) [noise=true , address=45.91.116.9:4264 , piece_id=VVK3P3Y4M5Y3V4FBCOUXB7R4EMYGV4Q7SCX6GBONJ7HHSAAL5CMQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:59.298343317 (636h3m59.169426362s) [noise=true , address=45.91.116.9:4264 , piece_id=VVK3P3Y4M5Y3V4FBCOUXB7R4EMYGV4Q7SCX6GBONJ7HHSAAL5CMQ, seg_pos={0 0}, upload=557348]: Long tail cancellation due to optimal threshold Jan 1 03:32:59.298498241 (636h3m59.169581286s) [noise=true , address=77.90.31.17:4773 , piece_id=YMMIFWC3IKTPKPM3XVPOVTLOPFJGIY6TNSUJQPWIMBGVKN6OWT7Q, seg_pos={0 0}, upload=557348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:59.298507059 (636h3m59.169590104s) [noise=true , address=77.90.31.17:4773 , piece_id=YMMIFWC3IKTPKPM3XVPOVTLOPFJGIY6TNSUJQPWIMBGVKN6OWT7Q, seg_pos={0 0}, upload=557348]: Long tail cancellation due to optimal threshold Jan 1 03:32:59.298525083 (636h3m59.169608118s) [seg_pos={0 0}, upload=557348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:32:59.298601798 (636h3m59.169684834s) [upload=557348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:32:59.298607400 (636h3m59.169690445s) [upload=557348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:32:59.596195152 (636h3m59.467278187s) [upload=557348]: Done uploading segments... Jan 1 03:32:59.596215400 (636h3m59.467298435s) [upload=557348]: Upload finished. err: <nil> Duration: 30.004700479s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X......X............................X.] duration: 3.883828911s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:33:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.883858386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXX] duration: 30.003225706s trace: https://jaeger.dev.storj.io/trace/5f996ce9d9bbf2fe FAIL (2026-01-01 03:32:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: KYZOR5QPXYNGF2QPALDVMMCKOWE67JN3GCGHCRCFPEFOQAGODBFA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 6AK7HHAD2I45VLTX7T2IHLAJCNIJJE6VMBNLSPENYMNKCTVQQRVQ) context deadline exceeded ======LOGS====== Jan 1 03:32:21.533126069 (636h3m21.404209104s) [upload=557340]: Starting upload Jan 1 03:32:21.533183809 (636h3m21.404266844s) [upload=557340]: Done starting upload Jan 1 03:32:21.533305360 (636h3m21.404388405s) [upload=557340]: Uploading segments... Jan 1 03:32:21.533429076 (636h3m21.404512121s) [upload=557340]: Got next segment. Inline: false Jan 1 03:32:21.533475845 (636h3m21.40455889s) [upload=557340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:32:21.533481676 (636h3m21.404564721s) [upload=557340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:32:21.779280165 (636h3m21.65036321s) [seg_pos={0 0}, upload=557340]: Begin upload segment... Jan 1 03:32:21.779373954 (636h3m21.650456989s) [seg_pos={0 0}, upload=557340]: Done begin upload segment. Jan 1 03:32:21.779471890 (636h3m21.650554935s) [noise=true , address=45.91.116.11:4120 , piece_id=KYZOR5QPXYNGF2QPALDVMMCKOWE67JN3GCGHCRCFPEFOQAGODBFA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.780649683 (636h3m21.651732728s) [noise=true , address=145.79.227.26:4653 , piece_id=NLQPCKTXXOHB6IOCIYO2EMBBH6O436ISSK7WFVF4T752XJMAH55Q, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.780710608 (636h3m21.651793654s) [noise=true , address=145.79.227.44:4193 , piece_id=CFF6YPJVIP7FLNF6VXBZTX5C6KAYARTBRYLJASE2YGD43OVPN4LA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.781433004 (636h3m21.652516049s) [noise=true , address=145.79.227.49:4339 , piece_id=TJW3ZHTOIQS2NLIYOQQLUEFEMNK3MTP4HFVH442JEB7DWRXENFTQ, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.781554156 (636h3m21.652637211s) [noise=true , address=145.79.227.36:4518 , piece_id=SDJPCHG4AFP6RLNN3OZ2RIHSYKWGYJA6ZK4ZGMXFVNTIY2IL6PMA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.782056361 (636h3m21.653139396s) [noise=true , address=145.79.227.15:4876 , piece_id=MIUFMGPY23NPR2Z4K4QVYDJDY3S5YHONS3YRMG6VNECKIYGM4KTA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.782066911 (636h3m21.653149946s) [noise=true , address=45.91.116.14:4189 , piece_id=IE4BVWR2A4WQFSUPM5Q2FFZHSSBS575G4VKOZJ7NBPXTD442CDJQ, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.782459008 (636h3m21.653542043s) [noise=true , address=145.79.227.48:4301 , piece_id=4SPOX6OMJ7PBI4GI3INKON7F5UYCASAHYCZ4ZZRLBNJLVPP5N52A, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.782513662 (636h3m21.653596707s) [noise=true , address=45.91.116.5:4297 , piece_id=ZPKLTUMT3WQ2ILHUU5XBR5HYKZKUWEUVF6ZTVXPO4YBXXK4P5A3A, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.782813423 (636h3m21.653896468s) [noise=true , address=77.90.31.19:4842 , piece_id=6AK7HHAD2I45VLTX7T2IHLAJCNIJJE6VMBNLSPENYMNKCTVQQRVQ, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.783429045 (636h3m21.65451208s) [noise=true , address=145.79.227.42:4122 , piece_id=NCYULIDBIUPKJ5PUZQH6JKZKQZOX4WDCDQUAQYQDH2UTL527TYCQ, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.783977630 (636h3m21.655060675s) [noise=true , address=145.79.227.38:4977 , piece_id=ASUYRO6S4ZCQXNPWQ5E5LIILESZAMH3YFUGKPLPFODJXR5QWKZPQ, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.784379925 (636h3m21.65546297s) [noise=true , address=145.79.227.14:4839 , piece_id=5W3RQKD25LDVZB3DBAJZUEKBITBAXIL2EEXXIM3WRAKCMHAQQVKA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.784673465 (636h3m21.6557565s) [noise=true , address=145.79.227.24:4583 , piece_id=6274FGXZGVND4OZ6J5IBIFHZEMIJ5WIPTDS3D4SJ6NLZQBINV36A, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.785140574 (636h3m21.656223619s) [noise=true , address=145.79.227.10:4401 , piece_id=B7JG6XH6ULIR4GM34QXQXPIWMADYHAGG5APXPYYCGXNEQ2UZV7FA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.786476729 (636h3m21.657559774s) [noise=true , address=145.79.227.17:4972 , piece_id=LDRLCVE5WL2UMRUQSCLQPLZCNVK3YR4CKEAYO27QLFOHZ3YLQDQA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:23.492415864 (636h3m23.363498909s) [upload=557340]: Next returned nil segment Jan 1 03:32:23.492442043 (636h3m23.363525078s) [upload=557340]: Waiting for error group managing segments... Jan 1 03:32:23.766492066 (636h3m23.637575101s) [noise=true , address=145.79.227.10:4401 , piece_id=B7JG6XH6ULIR4GM34QXQXPIWMADYHAGG5APXPYYCGXNEQ2UZV7FA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.782627920 (636h3m23.653710965s) [noise=true , address=145.79.227.42:4122 , piece_id=NCYULIDBIUPKJ5PUZQH6JKZKQZOX4WDCDQUAQYQDH2UTL527TYCQ, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.785317922 (636h3m23.656400967s) [noise=true , address=145.79.227.15:4876 , piece_id=MIUFMGPY23NPR2Z4K4QVYDJDY3S5YHONS3YRMG6VNECKIYGM4KTA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.785507564 (636h3m23.656590599s) [noise=true , address=145.79.227.48:4301 , piece_id=4SPOX6OMJ7PBI4GI3INKON7F5UYCASAHYCZ4ZZRLBNJLVPP5N52A, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.788764124 (636h3m23.659847159s) [noise=true , address=145.79.227.26:4653 , piece_id=NLQPCKTXXOHB6IOCIYO2EMBBH6O436ISSK7WFVF4T752XJMAH55Q, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.795590282 (636h3m23.666673328s) [noise=true , address=145.79.227.17:4972 , piece_id=LDRLCVE5WL2UMRUQSCLQPLZCNVK3YR4CKEAYO27QLFOHZ3YLQDQA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.809767047 (636h3m23.680850092s) [noise=true , address=145.79.227.38:4977 , piece_id=ASUYRO6S4ZCQXNPWQ5E5LIILESZAMH3YFUGKPLPFODJXR5QWKZPQ, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.815132945 (636h3m23.68621599s) [noise=true , address=145.79.227.14:4839 , piece_id=5W3RQKD25LDVZB3DBAJZUEKBITBAXIL2EEXXIM3WRAKCMHAQQVKA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.833498404 (636h3m23.704581449s) [noise=true , address=145.79.227.24:4583 , piece_id=6274FGXZGVND4OZ6J5IBIFHZEMIJ5WIPTDS3D4SJ6NLZQBINV36A, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.860339824 (636h3m23.731422869s) [noise=true , address=145.79.227.36:4518 , piece_id=SDJPCHG4AFP6RLNN3OZ2RIHSYKWGYJA6ZK4ZGMXFVNTIY2IL6PMA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.975885265 (636h3m23.8469683s) [noise=true , address=77.90.31.19:4842 , piece_id=6AK7HHAD2I45VLTX7T2IHLAJCNIJJE6VMBNLSPENYMNKCTVQQRVQ, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:24.136711647 (636h3m24.007794682s) [noise=true , address=45.91.116.14:4189 , piece_id=IE4BVWR2A4WQFSUPM5Q2FFZHSSBS575G4VKOZJ7NBPXTD442CDJQ, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:24.169079149 (636h3m24.040162184s) [noise=true , address=45.91.116.11:4120 , piece_id=KYZOR5QPXYNGF2QPALDVMMCKOWE67JN3GCGHCRCFPEFOQAGODBFA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:24.169109726 (636h3m24.040192771s) [seg_pos={0 0}, upload=557340]: Segment reached optimal threshold of 13 pieces. Jan 1 03:32:24.169358230 (636h3m24.040441285s) [noise=true , address=45.91.116.5:4297 , piece_id=ZPKLTUMT3WQ2ILHUU5XBR5HYKZKUWEUVF6ZTVXPO4YBXXK4P5A3A, seg_pos={0 0}, upload=557340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:24.169367537 (636h3m24.040450573s) [noise=true , address=45.91.116.5:4297 , piece_id=ZPKLTUMT3WQ2ILHUU5XBR5HYKZKUWEUVF6ZTVXPO4YBXXK4P5A3A, seg_pos={0 0}, upload=557340]: Long tail cancellation due to optimal threshold Jan 1 03:32:24.169486323 (636h3m24.040569368s) [noise=true , address=145.79.227.44:4193 , piece_id=CFF6YPJVIP7FLNF6VXBZTX5C6KAYARTBRYLJASE2YGD43OVPN4LA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:24.169544254 (636h3m24.040627289s) [noise=true , address=145.79.227.44:4193 , piece_id=CFF6YPJVIP7FLNF6VXBZTX5C6KAYARTBRYLJASE2YGD43OVPN4LA, seg_pos={0 0}, upload=557340]: Long tail cancellation due to optimal threshold Jan 1 03:32:24.169725509 (636h3m24.040808554s) [noise=true , address=145.79.227.49:4339 , piece_id=TJW3ZHTOIQS2NLIYOQQLUEFEMNK3MTP4HFVH442JEB7DWRXENFTQ, seg_pos={0 0}, upload=557340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:24.169735307 (636h3m24.040818352s) [noise=true , address=145.79.227.49:4339 , piece_id=TJW3ZHTOIQS2NLIYOQQLUEFEMNK3MTP4HFVH442JEB7DWRXENFTQ, seg_pos={0 0}, upload=557340]: Long tail cancellation due to optimal threshold Jan 1 03:32:24.169762139 (636h3m24.040845174s) [seg_pos={0 0}, upload=557340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:32:24.169888510 (636h3m24.040971555s) [upload=557340]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:32:24.169895583 (636h3m24.040978628s) [upload=557340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:32:24.466277248 (636h3m24.337360283s) [upload=557340]: Done uploading segments... Jan 1 03:32:24.466294611 (636h3m24.337377646s) [upload=557340]: Upload finished. err: <nil> Duration: 30.003244511s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X......X............................X] duration: 5.001216178s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:32:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001240835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XX] duration: 30.003870766s trace: https://jaeger.dev.storj.io/trace/27e7b1a304ec007a FAIL (2026-01-01 03:32:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: HEAZI373ADYHLWCH64NG55TMGOC5QD2VJSFH5LIBEOVQAJSNQOXQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 7OFTTAHEMU4YGOYH4AAVT57QPZFQ3GB2CIW6374MV3UOYGNFMLTQ) context deadline exceeded ======LOGS====== Jan 1 03:31:48.605861455 (636h2m48.4769445s) [upload=557331]: Starting upload Jan 1 03:31:48.605945095 (636h2m48.47702813s) [upload=557331]: Done starting upload Jan 1 03:31:48.606120018 (636h2m48.477203053s) [upload=557331]: Uploading segments... Jan 1 03:31:48.606250606 (636h2m48.477333641s) [upload=557331]: Got next segment. Inline: false Jan 1 03:31:48.606283599 (636h2m48.477366634s) [upload=557331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:31:48.606288388 (636h2m48.477371423s) [upload=557331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:31:48.851684150 (636h2m48.722767195s) [seg_pos={0 0}, upload=557331]: Begin upload segment... Jan 1 03:31:48.851741389 (636h2m48.722824434s) [seg_pos={0 0}, upload=557331]: Done begin upload segment. Jan 1 03:31:48.851777037 (636h2m48.722860082s) [noise=true , address=45.91.116.14:4192 , piece_id=HEAZI373ADYHLWCH64NG55TMGOC5QD2VJSFH5LIBEOVQAJSNQOXQ, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.851882367 (636h2m48.722965412s) [noise=true , address=145.79.227.48:4303 , piece_id=YWOGX5KOTNIS272TFSWAI26S7NPT35622OYGU6DIHHFOZK7DSRIA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.852630432 (636h2m48.723713467s) [noise=true , address=45.91.116.11:4117 , piece_id=7OFTTAHEMU4YGOYH4AAVT57QPZFQ3GB2CIW6374MV3UOYGNFMLTQ, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.852718369 (636h2m48.723801414s) [noise=true , address=45.91.116.9:4264 , piece_id=4UUZVQMLH6IX7V2GJ7J2W7TJLL3YL6CGM6TE7KSA47GKBJSFHCAA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.853214414 (636h2m48.724297459s) [noise=true , address=145.79.227.31:4799 , piece_id=YBIZK2CKC5J72G2UPTDOX6CFI6GYKACLBUJIVTWBCMNRCFESVIDA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.853786924 (636h2m48.724869959s) [noise=true , address=45.91.116.5:4298 , piece_id=33EEMPOVT3YIAVOI4Y5HDIERAPW23INCUKM7U7AJE4QUHPS4W5ZA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.854350647 (636h2m48.725433682s) [noise=true , address=145.79.227.46:4601 , piece_id=MJ4XDSFSFZWNFBR7ZNP6UMPK6XESIV6RAYAIQVOU4I6BEOBEWBKQ, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.854424257 (636h2m48.725507292s) [noise=true , address=45.91.116.10:4082 , piece_id=HTYHF337UCIXD6IU7ARUB2OM7DE22D4PURZZHT26QX3KEJBYB63A, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.855391830 (636h2m48.726474875s) [noise=true , address=145.79.227.6:4259 , piece_id=OVEKDISJ6JUL7CVF2XPRCZ27ILNFWOF22T6R7HV6N7HHRAYTN27Q, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.855607961 (636h2m48.726691006s) [noise=true , address=145.79.227.22:4511 , piece_id=JRDU5A3LB4SXFZVFYB3HEIRCPD2X4YNKGM64UWGSL7APS3CBGWPA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.856570665 (636h2m48.7276537s) [noise=true , address=45.91.116.6:4974 , piece_id=4L2LSQNG773O6BDMRHRPVONEQT45NZH3V5V4WDPA5UB5RPA52C3Q, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.857436083 (636h2m48.728519118s) [noise=true , address=145.79.227.15:4875 , piece_id=VNTJE3D43O6K4OIFMYMK4IQD76F7YLA7ZESWAYFQLHQU42B2AVFA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.858042507 (636h2m48.729125552s) [noise=true , address=145.79.227.51:4411 , piece_id=CD6RATWTKYQMK7H2QUPJXYED2GN5247OFJP4YDF5NKL3VNSNSWPA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.858702133 (636h2m48.729785178s) [noise=true , address=145.79.227.23:4547 , piece_id=6WQ4QYRTB3XKQ2N4CT7XXFAXXW34SLWAYHIGLQLPZEW2AKGGYZBA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.861019756 (636h2m48.732102791s) [noise=true , address=145.79.227.47:4266 , piece_id=S2IADG6ZKUOC2MCNGTDNLWKTGVLXJBSIZACZMMZG32SSQVWHYRMQ, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.862034197 (636h2m48.733117232s) [noise=true , address=77.90.31.16:4735 , piece_id=ZQUXSBAAAMRTY5VH2T4RKPRHBHOJ74KBYAX2ROP53K6FT25SRG7Q, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:50.450701976 (636h2m50.321785021s) [upload=557331]: Next returned nil segment Jan 1 03:31:50.450715532 (636h2m50.321798577s) [upload=557331]: Waiting for error group managing segments... Jan 1 03:31:50.832638738 (636h2m50.703721783s) [noise=true , address=145.79.227.23:4547 , piece_id=6WQ4QYRTB3XKQ2N4CT7XXFAXXW34SLWAYHIGLQLPZEW2AKGGYZBA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.853744105 (636h2m50.72482715s) [noise=true , address=145.79.227.22:4511 , piece_id=JRDU5A3LB4SXFZVFYB3HEIRCPD2X4YNKGM64UWGSL7APS3CBGWPA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.854304953 (636h2m50.725387999s) [noise=true , address=145.79.227.31:4799 , piece_id=YBIZK2CKC5J72G2UPTDOX6CFI6GYKACLBUJIVTWBCMNRCFESVIDA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.858543283 (636h2m50.729626328s) [noise=true , address=145.79.227.6:4259 , piece_id=OVEKDISJ6JUL7CVF2XPRCZ27ILNFWOF22T6R7HV6N7HHRAYTN27Q, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.858656919 (636h2m50.729739964s) [noise=true , address=145.79.227.47:4266 , piece_id=S2IADG6ZKUOC2MCNGTDNLWKTGVLXJBSIZACZMMZG32SSQVWHYRMQ, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.865560475 (636h2m50.73664352s) [noise=true , address=145.79.227.48:4303 , piece_id=YWOGX5KOTNIS272TFSWAI26S7NPT35622OYGU6DIHHFOZK7DSRIA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.928801002 (636h2m50.799884067s) [noise=true , address=145.79.227.15:4875 , piece_id=VNTJE3D43O6K4OIFMYMK4IQD76F7YLA7ZESWAYFQLHQU42B2AVFA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.931695644 (636h2m50.802778689s) [noise=true , address=145.79.227.51:4411 , piece_id=CD6RATWTKYQMK7H2QUPJXYED2GN5247OFJP4YDF5NKL3VNSNSWPA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.932461772 (636h2m50.803544807s) [noise=true , address=145.79.227.46:4601 , piece_id=MJ4XDSFSFZWNFBR7ZNP6UMPK6XESIV6RAYAIQVOU4I6BEOBEWBKQ, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:51.275350643 (636h2m51.146433688s) [noise=true , address=45.91.116.9:4264 , piece_id=4UUZVQMLH6IX7V2GJ7J2W7TJLL3YL6CGM6TE7KSA47GKBJSFHCAA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:51.283898479 (636h2m51.154981524s) [noise=true , address=45.91.116.6:4974 , piece_id=4L2LSQNG773O6BDMRHRPVONEQT45NZH3V5V4WDPA5UB5RPA52C3Q, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:51.497298367 (636h2m51.368381402s) [noise=true , address=45.91.116.14:4192 , piece_id=HEAZI373ADYHLWCH64NG55TMGOC5QD2VJSFH5LIBEOVQAJSNQOXQ, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:51.526348722 (636h2m51.397431767s) [noise=true , address=45.91.116.11:4117 , piece_id=7OFTTAHEMU4YGOYH4AAVT57QPZFQ3GB2CIW6374MV3UOYGNFMLTQ, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:51.526375443 (636h2m51.397458488s) [seg_pos={0 0}, upload=557331]: Segment reached optimal threshold of 13 pieces. Jan 1 03:31:51.526610039 (636h2m51.397693074s) [noise=true , address=45.91.116.10:4082 , piece_id=HTYHF337UCIXD6IU7ARUB2OM7DE22D4PURZZHT26QX3KEJBYB63A, seg_pos={0 0}, upload=557331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 1 03:31:51.526620600 (636h2m51.397703645s) [noise=true , address=45.91.116.10:4082 , piece_id=HTYHF337UCIXD6IU7ARUB2OM7DE22D4PURZZHT26QX3KEJBYB63A, seg_pos={0 0}, upload=557331]: Long tail cancellation due to optimal threshold Jan 1 03:31:51.526631481 (636h2m51.397714516s) [noise=true , address=77.90.31.16:4735 , piece_id=ZQUXSBAAAMRTY5VH2T4RKPRHBHOJ74KBYAX2ROP53K6FT25SRG7Q, seg_pos={0 0}, upload=557331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jan 1 03:31:51.526636811 (636h2m51.397719856s) [noise=true , address=77.90.31.16:4735 , piece_id=ZQUXSBAAAMRTY5VH2T4RKPRHBHOJ74KBYAX2ROP53K6FT25SRG7Q, seg_pos={0 0}, upload=557331]: Long tail cancellation due to optimal threshold Jan 1 03:31:51.526769023 (636h2m51.397852068s) [noise=true , address=45.91.116.5:4298 , piece_id=33EEMPOVT3YIAVOI4Y5HDIERAPW23INCUKM7U7AJE4QUHPS4W5ZA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:49332->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:49332->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49332->45.91.116.5:4298: use of closed network connection Jan 1 03:31:51.526792457 (636h2m51.397875493s) [noise=true , address=45.91.116.5:4298 , piece_id=33EEMPOVT3YIAVOI4Y5HDIERAPW23INCUKM7U7AJE4QUHPS4W5ZA, seg_pos={0 0}, upload=557331]: Long tail cancellation due to optimal threshold Jan 1 03:31:51.526810060 (636h2m51.397893096s) [seg_pos={0 0}, upload=557331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:31:51.526945388 (636h2m51.398028423s) [upload=557331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:31:51.526975907 (636h2m51.398058952s) [upload=557331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:31:51.813606695 (636h2m51.68468973s) [upload=557331]: Done uploading segments... Jan 1 03:31:51.813655197 (636h2m51.684738242s) [upload=557331]: Upload finished. err: <nil> Duration: 30.003895842s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)