Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..XX..X.....X.........X....XXX..XX....X...XXXX...X..X......] duration: 5.901124737s trace: https://jaeger.dev.storj.io/trace/2aaaa63992cda295 FAIL (2025-12-27 19:34:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA) context deadline exceeded ======LOGS====== Dec 27 19:28:55.244009263 (531h59m55.115092298s) [upload=466566]: Starting upload Dec 27 19:28:55.244080528 (531h59m55.115163563s) [upload=466566]: Done starting upload Dec 27 19:28:55.244121197 (531h59m55.115204242s) [upload=466566]: Uploading segments... Dec 27 19:28:55.244264059 (531h59m55.115347094s) [upload=466566]: Got next segment. Inline: false Dec 27 19:28:55.244293524 (531h59m55.115376559s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:28:55.244297613 (531h59m55.115380648s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:28:55.492122396 (531h59m55.363205442s) [seg_pos={0 0}, upload=466566]: Begin upload segment... Dec 27 19:28:55.492171831 (531h59m55.363254866s) [seg_pos={0 0}, upload=466566]: Done begin upload segment. Dec 27 19:28:55.492210044 (531h59m55.363293089s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492276951 (531h59m55.363359996s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492788987 (531h59m55.363872022s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493290882 (531h59m55.364373917s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493659594 (531h59m55.364742629s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493727313 (531h59m55.364810348s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494031863 (531h59m55.365114898s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494347785 (531h59m55.36543082s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494650821 (531h59m55.365733856s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494692571 (531h59m55.365775606s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495048329 (531h59m55.366131364s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495131587 (531h59m55.366214632s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495368609 (531h59m55.366451644s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495703786 (531h59m55.366786821s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495714557 (531h59m55.366797592s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.496163001 (531h59m55.367246036s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:57.194267661 (531h59m57.065350706s) [upload=466566]: Next returned nil segment Dec 27 19:28:57.194283752 (531h59m57.065366787s) [upload=466566]: Waiting for error group managing segments... Dec 27 19:28:57.485520010 (531h59m57.356603045s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.486304192 (531h59m57.357387227s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.493980710 (531h59m57.365063775s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.498600788 (531h59m57.369683823s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.503839474 (531h59m57.374922509s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.510535855 (531h59m57.38161889s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.515721218 (531h59m57.386804254s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.523984714 (531h59m57.395067759s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.527884330 (531h59m57.398967365s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.686365167 (531h59m57.557448202s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.748971160 (531h59m57.620054195s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.838954796 (531h59m57.710037831s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844423729 (531h59m57.715506764s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844443177 (531h59m57.715526222s) [seg_pos={0 0}, upload=466566]: Segment reached optimal threshold of 13 pieces. Dec 27 19:28:57.844708442 (531h59m57.715791477s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844716407 (531h59m57.715799442s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844837908 (531h59m57.715920963s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844874618 (531h59m57.715957653s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844970330 (531h59m57.716053375s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844979608 (531h59m57.716062643s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844999486 (531h59m57.716082521s) [seg_pos={0 0}, upload=466566]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:28:57.845101981 (531h59m57.716185016s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:28:57.845120386 (531h59m57.716203421s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:28:58.136270631 (531h59m58.007353696s) [upload=466566]: Done uploading segments... Dec 27 19:28:58.136292552 (531h59m58.007375587s) [upload=466566]: Upload finished. err: <nil> Duration: 5.901154973s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.465554675s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:33:38 UTC) orbiter warning: context deadline exceeded Duration: 3.46558345s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX..XX..X.....X.........X....XXX..XX....X...XXXX...X..X.....] duration: 5.82928411s trace: https://jaeger.dev.storj.io/trace/2aaaa63992cda295 FAIL (2025-12-27 19:32:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA) context deadline exceeded ======LOGS====== Dec 27 19:28:55.244009263 (531h59m55.115092298s) [upload=466566]: Starting upload Dec 27 19:28:55.244080528 (531h59m55.115163563s) [upload=466566]: Done starting upload Dec 27 19:28:55.244121197 (531h59m55.115204242s) [upload=466566]: Uploading segments... Dec 27 19:28:55.244264059 (531h59m55.115347094s) [upload=466566]: Got next segment. Inline: false Dec 27 19:28:55.244293524 (531h59m55.115376559s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:28:55.244297613 (531h59m55.115380648s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:28:55.492122396 (531h59m55.363205442s) [seg_pos={0 0}, upload=466566]: Begin upload segment... Dec 27 19:28:55.492171831 (531h59m55.363254866s) [seg_pos={0 0}, upload=466566]: Done begin upload segment. Dec 27 19:28:55.492210044 (531h59m55.363293089s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492276951 (531h59m55.363359996s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492788987 (531h59m55.363872022s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493290882 (531h59m55.364373917s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493659594 (531h59m55.364742629s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493727313 (531h59m55.364810348s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494031863 (531h59m55.365114898s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494347785 (531h59m55.36543082s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494650821 (531h59m55.365733856s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494692571 (531h59m55.365775606s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495048329 (531h59m55.366131364s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495131587 (531h59m55.366214632s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495368609 (531h59m55.366451644s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495703786 (531h59m55.366786821s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495714557 (531h59m55.366797592s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.496163001 (531h59m55.367246036s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:57.194267661 (531h59m57.065350706s) [upload=466566]: Next returned nil segment Dec 27 19:28:57.194283752 (531h59m57.065366787s) [upload=466566]: Waiting for error group managing segments... Dec 27 19:28:57.485520010 (531h59m57.356603045s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.486304192 (531h59m57.357387227s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.493980710 (531h59m57.365063775s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.498600788 (531h59m57.369683823s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.503839474 (531h59m57.374922509s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.510535855 (531h59m57.38161889s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.515721218 (531h59m57.386804254s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.523984714 (531h59m57.395067759s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.527884330 (531h59m57.398967365s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.686365167 (531h59m57.557448202s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.748971160 (531h59m57.620054195s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.838954796 (531h59m57.710037831s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844423729 (531h59m57.715506764s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844443177 (531h59m57.715526222s) [seg_pos={0 0}, upload=466566]: Segment reached optimal threshold of 13 pieces. Dec 27 19:28:57.844708442 (531h59m57.715791477s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844716407 (531h59m57.715799442s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844837908 (531h59m57.715920963s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844874618 (531h59m57.715957653s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844970330 (531h59m57.716053375s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844979608 (531h59m57.716062643s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844999486 (531h59m57.716082521s) [seg_pos={0 0}, upload=466566]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:28:57.845101981 (531h59m57.716185016s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:28:57.845120386 (531h59m57.716203421s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:28:58.136270631 (531h59m58.007353696s) [upload=466566]: Done uploading segments... Dec 27 19:28:58.136292552 (531h59m58.007375587s) [upload=466566]: Upload finished. err: <nil> Duration: 5.829312163s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.530595587s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:32:34 UTC) orbiter warning: context deadline exceeded Duration: 3.530634363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX..XX..X.....X.........X....XXX..XX....X...XXXX...X..X....] duration: 6.222867285s trace: https://jaeger.dev.storj.io/trace/2aaaa63992cda295 FAIL (2025-12-27 19:32:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA) context deadline exceeded ======LOGS====== Dec 27 19:28:55.244009263 (531h59m55.115092298s) [upload=466566]: Starting upload Dec 27 19:28:55.244080528 (531h59m55.115163563s) [upload=466566]: Done starting upload Dec 27 19:28:55.244121197 (531h59m55.115204242s) [upload=466566]: Uploading segments... Dec 27 19:28:55.244264059 (531h59m55.115347094s) [upload=466566]: Got next segment. Inline: false Dec 27 19:28:55.244293524 (531h59m55.115376559s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:28:55.244297613 (531h59m55.115380648s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:28:55.492122396 (531h59m55.363205442s) [seg_pos={0 0}, upload=466566]: Begin upload segment... Dec 27 19:28:55.492171831 (531h59m55.363254866s) [seg_pos={0 0}, upload=466566]: Done begin upload segment. Dec 27 19:28:55.492210044 (531h59m55.363293089s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492276951 (531h59m55.363359996s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492788987 (531h59m55.363872022s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493290882 (531h59m55.364373917s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493659594 (531h59m55.364742629s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493727313 (531h59m55.364810348s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494031863 (531h59m55.365114898s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494347785 (531h59m55.36543082s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494650821 (531h59m55.365733856s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494692571 (531h59m55.365775606s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495048329 (531h59m55.366131364s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495131587 (531h59m55.366214632s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495368609 (531h59m55.366451644s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495703786 (531h59m55.366786821s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495714557 (531h59m55.366797592s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.496163001 (531h59m55.367246036s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:57.194267661 (531h59m57.065350706s) [upload=466566]: Next returned nil segment Dec 27 19:28:57.194283752 (531h59m57.065366787s) [upload=466566]: Waiting for error group managing segments... Dec 27 19:28:57.485520010 (531h59m57.356603045s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.486304192 (531h59m57.357387227s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.493980710 (531h59m57.365063775s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.498600788 (531h59m57.369683823s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.503839474 (531h59m57.374922509s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.510535855 (531h59m57.38161889s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.515721218 (531h59m57.386804254s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.523984714 (531h59m57.395067759s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.527884330 (531h59m57.398967365s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.686365167 (531h59m57.557448202s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.748971160 (531h59m57.620054195s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.838954796 (531h59m57.710037831s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844423729 (531h59m57.715506764s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844443177 (531h59m57.715526222s) [seg_pos={0 0}, upload=466566]: Segment reached optimal threshold of 13 pieces. Dec 27 19:28:57.844708442 (531h59m57.715791477s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844716407 (531h59m57.715799442s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844837908 (531h59m57.715920963s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844874618 (531h59m57.715957653s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844970330 (531h59m57.716053375s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844979608 (531h59m57.716062643s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844999486 (531h59m57.716082521s) [seg_pos={0 0}, upload=466566]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:28:57.845101981 (531h59m57.716185016s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:28:57.845120386 (531h59m57.716203421s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:28:58.136270631 (531h59m58.007353696s) [upload=466566]: Done uploading segments... Dec 27 19:28:58.136292552 (531h59m58.007375587s) [upload=466566]: Upload finished. err: <nil> Duration: 6.222898724s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.608066834s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:31:51 UTC) orbiter warning: context deadline exceeded Duration: 3.60809618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XX..XX..X.....X.........X....XXX..XX....X...XXXX...X..X...] duration: 5.8979125s trace: https://jaeger.dev.storj.io/trace/2aaaa63992cda295 FAIL (2025-12-27 19:31:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA) context deadline exceeded ======LOGS====== Dec 27 19:28:55.244009263 (531h59m55.115092298s) [upload=466566]: Starting upload Dec 27 19:28:55.244080528 (531h59m55.115163563s) [upload=466566]: Done starting upload Dec 27 19:28:55.244121197 (531h59m55.115204242s) [upload=466566]: Uploading segments... Dec 27 19:28:55.244264059 (531h59m55.115347094s) [upload=466566]: Got next segment. Inline: false Dec 27 19:28:55.244293524 (531h59m55.115376559s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:28:55.244297613 (531h59m55.115380648s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:28:55.492122396 (531h59m55.363205442s) [seg_pos={0 0}, upload=466566]: Begin upload segment... Dec 27 19:28:55.492171831 (531h59m55.363254866s) [seg_pos={0 0}, upload=466566]: Done begin upload segment. Dec 27 19:28:55.492210044 (531h59m55.363293089s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492276951 (531h59m55.363359996s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492788987 (531h59m55.363872022s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493290882 (531h59m55.364373917s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493659594 (531h59m55.364742629s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493727313 (531h59m55.364810348s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494031863 (531h59m55.365114898s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494347785 (531h59m55.36543082s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494650821 (531h59m55.365733856s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494692571 (531h59m55.365775606s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495048329 (531h59m55.366131364s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495131587 (531h59m55.366214632s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495368609 (531h59m55.366451644s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495703786 (531h59m55.366786821s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495714557 (531h59m55.366797592s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.496163001 (531h59m55.367246036s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:57.194267661 (531h59m57.065350706s) [upload=466566]: Next returned nil segment Dec 27 19:28:57.194283752 (531h59m57.065366787s) [upload=466566]: Waiting for error group managing segments... Dec 27 19:28:57.485520010 (531h59m57.356603045s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.486304192 (531h59m57.357387227s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.493980710 (531h59m57.365063775s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.498600788 (531h59m57.369683823s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.503839474 (531h59m57.374922509s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.510535855 (531h59m57.38161889s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.515721218 (531h59m57.386804254s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.523984714 (531h59m57.395067759s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.527884330 (531h59m57.398967365s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.686365167 (531h59m57.557448202s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.748971160 (531h59m57.620054195s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.838954796 (531h59m57.710037831s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844423729 (531h59m57.715506764s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844443177 (531h59m57.715526222s) [seg_pos={0 0}, upload=466566]: Segment reached optimal threshold of 13 pieces. Dec 27 19:28:57.844708442 (531h59m57.715791477s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844716407 (531h59m57.715799442s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844837908 (531h59m57.715920963s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844874618 (531h59m57.715957653s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844970330 (531h59m57.716053375s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844979608 (531h59m57.716062643s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844999486 (531h59m57.716082521s) [seg_pos={0 0}, upload=466566]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:28:57.845101981 (531h59m57.716185016s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:28:57.845120386 (531h59m57.716203421s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:28:58.136270631 (531h59m58.007353696s) [upload=466566]: Done uploading segments... Dec 27 19:28:58.136292552 (531h59m58.007375587s) [upload=466566]: Upload finished. err: <nil> Duration: 5.897945863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...XX..XX..X.....X.........X....XXX..XX....X...XXXX...X..X..] duration: 6.744951183s trace: https://jaeger.dev.storj.io/trace/2aaaa63992cda295 FAIL (2025-12-27 19:31:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA) context deadline exceeded ======LOGS====== Dec 27 19:28:55.244009263 (531h59m55.115092298s) [upload=466566]: Starting upload Dec 27 19:28:55.244080528 (531h59m55.115163563s) [upload=466566]: Done starting upload Dec 27 19:28:55.244121197 (531h59m55.115204242s) [upload=466566]: Uploading segments... Dec 27 19:28:55.244264059 (531h59m55.115347094s) [upload=466566]: Got next segment. Inline: false Dec 27 19:28:55.244293524 (531h59m55.115376559s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:28:55.244297613 (531h59m55.115380648s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:28:55.492122396 (531h59m55.363205442s) [seg_pos={0 0}, upload=466566]: Begin upload segment... Dec 27 19:28:55.492171831 (531h59m55.363254866s) [seg_pos={0 0}, upload=466566]: Done begin upload segment. Dec 27 19:28:55.492210044 (531h59m55.363293089s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492276951 (531h59m55.363359996s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492788987 (531h59m55.363872022s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493290882 (531h59m55.364373917s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493659594 (531h59m55.364742629s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493727313 (531h59m55.364810348s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494031863 (531h59m55.365114898s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494347785 (531h59m55.36543082s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494650821 (531h59m55.365733856s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494692571 (531h59m55.365775606s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495048329 (531h59m55.366131364s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495131587 (531h59m55.366214632s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495368609 (531h59m55.366451644s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495703786 (531h59m55.366786821s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495714557 (531h59m55.366797592s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.496163001 (531h59m55.367246036s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:57.194267661 (531h59m57.065350706s) [upload=466566]: Next returned nil segment Dec 27 19:28:57.194283752 (531h59m57.065366787s) [upload=466566]: Waiting for error group managing segments... Dec 27 19:28:57.485520010 (531h59m57.356603045s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.486304192 (531h59m57.357387227s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.493980710 (531h59m57.365063775s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.498600788 (531h59m57.369683823s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.503839474 (531h59m57.374922509s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.510535855 (531h59m57.38161889s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.515721218 (531h59m57.386804254s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.523984714 (531h59m57.395067759s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.527884330 (531h59m57.398967365s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.686365167 (531h59m57.557448202s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.748971160 (531h59m57.620054195s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.838954796 (531h59m57.710037831s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844423729 (531h59m57.715506764s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844443177 (531h59m57.715526222s) [seg_pos={0 0}, upload=466566]: Segment reached optimal threshold of 13 pieces. Dec 27 19:28:57.844708442 (531h59m57.715791477s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844716407 (531h59m57.715799442s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844837908 (531h59m57.715920963s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844874618 (531h59m57.715957653s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844970330 (531h59m57.716053375s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844979608 (531h59m57.716062643s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844999486 (531h59m57.716082521s) [seg_pos={0 0}, upload=466566]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:28:57.845101981 (531h59m57.716185016s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:28:57.845120386 (531h59m57.716203421s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:28:58.136270631 (531h59m58.007353696s) [upload=466566]: Done uploading segments... Dec 27 19:28:58.136292552 (531h59m58.007375587s) [upload=466566]: Upload finished. err: <nil> Duration: 6.74498183s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.549207928s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:30:52 UTC) orbiter warning: context deadline exceeded Duration: 3.549249968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [....XX..XX..X.....X.........X....XXX..XX....X...XXXX...X..X.] duration: 6.03106961s trace: https://jaeger.dev.storj.io/trace/2aaaa63992cda295 FAIL (2025-12-27 19:30:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA) context deadline exceeded ======LOGS====== Dec 27 19:28:55.244009263 (531h59m55.115092298s) [upload=466566]: Starting upload Dec 27 19:28:55.244080528 (531h59m55.115163563s) [upload=466566]: Done starting upload Dec 27 19:28:55.244121197 (531h59m55.115204242s) [upload=466566]: Uploading segments... Dec 27 19:28:55.244264059 (531h59m55.115347094s) [upload=466566]: Got next segment. Inline: false Dec 27 19:28:55.244293524 (531h59m55.115376559s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:28:55.244297613 (531h59m55.115380648s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:28:55.492122396 (531h59m55.363205442s) [seg_pos={0 0}, upload=466566]: Begin upload segment... Dec 27 19:28:55.492171831 (531h59m55.363254866s) [seg_pos={0 0}, upload=466566]: Done begin upload segment. Dec 27 19:28:55.492210044 (531h59m55.363293089s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492276951 (531h59m55.363359996s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492788987 (531h59m55.363872022s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493290882 (531h59m55.364373917s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493659594 (531h59m55.364742629s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493727313 (531h59m55.364810348s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494031863 (531h59m55.365114898s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494347785 (531h59m55.36543082s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494650821 (531h59m55.365733856s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494692571 (531h59m55.365775606s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495048329 (531h59m55.366131364s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495131587 (531h59m55.366214632s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495368609 (531h59m55.366451644s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495703786 (531h59m55.366786821s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495714557 (531h59m55.366797592s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.496163001 (531h59m55.367246036s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:57.194267661 (531h59m57.065350706s) [upload=466566]: Next returned nil segment Dec 27 19:28:57.194283752 (531h59m57.065366787s) [upload=466566]: Waiting for error group managing segments... Dec 27 19:28:57.485520010 (531h59m57.356603045s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.486304192 (531h59m57.357387227s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.493980710 (531h59m57.365063775s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.498600788 (531h59m57.369683823s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.503839474 (531h59m57.374922509s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.510535855 (531h59m57.38161889s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.515721218 (531h59m57.386804254s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.523984714 (531h59m57.395067759s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.527884330 (531h59m57.398967365s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.686365167 (531h59m57.557448202s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.748971160 (531h59m57.620054195s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.838954796 (531h59m57.710037831s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844423729 (531h59m57.715506764s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844443177 (531h59m57.715526222s) [seg_pos={0 0}, upload=466566]: Segment reached optimal threshold of 13 pieces. Dec 27 19:28:57.844708442 (531h59m57.715791477s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844716407 (531h59m57.715799442s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844837908 (531h59m57.715920963s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844874618 (531h59m57.715957653s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844970330 (531h59m57.716053375s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844979608 (531h59m57.716062643s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844999486 (531h59m57.716082521s) [seg_pos={0 0}, upload=466566]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:28:57.845101981 (531h59m57.716185016s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:28:57.845120386 (531h59m57.716203421s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:28:58.136270631 (531h59m58.007353696s) [upload=466566]: Done uploading segments... Dec 27 19:28:58.136292552 (531h59m58.007375587s) [upload=466566]: Upload finished. err: <nil> Duration: 6.031105538s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.425325883s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:29:51 UTC) orbiter warning: context deadline exceeded Duration: 3.425344468s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....XX..XX..X.....X.........X....XXX..XX....X...XXXX...X..X] duration: 30.006036045s trace: https://jaeger.dev.storj.io/trace/2aaaa63992cda295 FAIL (2025-12-27 19:29:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA) context deadline exceeded ======LOGS====== Dec 27 19:28:55.244009263 (531h59m55.115092298s) [upload=466566]: Starting upload Dec 27 19:28:55.244080528 (531h59m55.115163563s) [upload=466566]: Done starting upload Dec 27 19:28:55.244121197 (531h59m55.115204242s) [upload=466566]: Uploading segments... Dec 27 19:28:55.244264059 (531h59m55.115347094s) [upload=466566]: Got next segment. Inline: false Dec 27 19:28:55.244293524 (531h59m55.115376559s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:28:55.244297613 (531h59m55.115380648s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:28:55.492122396 (531h59m55.363205442s) [seg_pos={0 0}, upload=466566]: Begin upload segment... Dec 27 19:28:55.492171831 (531h59m55.363254866s) [seg_pos={0 0}, upload=466566]: Done begin upload segment. Dec 27 19:28:55.492210044 (531h59m55.363293089s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492276951 (531h59m55.363359996s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.492788987 (531h59m55.363872022s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493290882 (531h59m55.364373917s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493659594 (531h59m55.364742629s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.493727313 (531h59m55.364810348s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494031863 (531h59m55.365114898s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494347785 (531h59m55.36543082s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494650821 (531h59m55.365733856s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.494692571 (531h59m55.365775606s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495048329 (531h59m55.366131364s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495131587 (531h59m55.366214632s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495368609 (531h59m55.366451644s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495703786 (531h59m55.366786821s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.495714557 (531h59m55.366797592s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:55.496163001 (531h59m55.367246036s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Uploading piece... Dec 27 19:28:57.194267661 (531h59m57.065350706s) [upload=466566]: Next returned nil segment Dec 27 19:28:57.194283752 (531h59m57.065366787s) [upload=466566]: Waiting for error group managing segments... Dec 27 19:28:57.485520010 (531h59m57.356603045s) [noise=true , address=145.79.227.18:4007 , piece_id=SG5ATVS7PQU3IE4O6W6KWSYNVMH2QHZEOOUYZJOPWEWAVEUKQPIQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.486304192 (531h59m57.357387227s) [noise=true , address=145.79.227.46:4601 , piece_id=NW4GRNY636HW7747URNJ7NPQ7KBFICDRADLABJEFWL7J7IJYIHSQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.493980710 (531h59m57.365063775s) [noise=true , address=145.79.227.48:4302 , piece_id=7EIIAINYONJKRYIEHA2V3IS2HGLVHBPN3QMYFNLLKVEEUIVVDTAQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.498600788 (531h59m57.369683823s) [noise=true , address=145.79.227.37:4566 , piece_id=76HEZO7NHHLR3V2V3IRXSRK5Y643KO5UB4UJFR7264N3MLC6LA7A, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.503839474 (531h59m57.374922509s) [noise=true , address=145.79.227.50:4374 , piece_id=UAHLUHZTNH24DI234XKHCUFNRETYPMP6X34UZUPKFMQFAICZY3XA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.510535855 (531h59m57.38161889s) [noise=true , address=145.79.227.11:4732 , piece_id=BQOP6K66Q3YFKFQE3HC6SSTYU5WEFIFF7VDRFC5A7PKGHONE7REQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.515721218 (531h59m57.386804254s) [noise=true , address=145.79.227.14:4839 , piece_id=DDNN2RU44IAOSMG5XAR6IFPDTAUJMV53W3PZJPIACCGLTLT2TXUQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.523984714 (531h59m57.395067759s) [noise=true , address=145.79.227.17:4972 , piece_id=4AK5SUNXR5DZ3MYS5JDVQVH4I5NRPJ5OFPNDF5U7KH3H64DPSHKQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.527884330 (531h59m57.398967365s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUDMB4QY4Q5NVA4OB2NY3RPEZUXX276C4ZHF3IADAHMJEJ55AZPA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.686365167 (531h59m57.557448202s) [noise=true , address=77.90.31.19:4842 , piece_id=3K77B4LAGP3SIYWAX4W73KZHL6DVGA67N7KZACOHQKBKZYJWRCLA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.748971160 (531h59m57.620054195s) [noise=true , address=77.90.31.17:4770 , piece_id=XLULOFNGOMEAC2KNRX7LXQFUNYLIVSYPWEE7PD7MYCBEELZREETA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.838954796 (531h59m57.710037831s) [noise=true , address=45.91.116.10:4083 , piece_id=OT2NMXQDWS47TYBZDNCTFXWDKJQOX2HZ4C3SJUSMJKNTIP5PHQEA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844423729 (531h59m57.715506764s) [noise=true , address=45.91.116.9:4261 , piece_id=TUMU76DWLMRMCAMG6IY63WEPYQJYIGFLV2L4LEXMEGKWNTFEKEDQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: <nil> Dec 27 19:28:57.844443177 (531h59m57.715526222s) [seg_pos={0 0}, upload=466566]: Segment reached optimal threshold of 13 pieces. Dec 27 19:28:57.844708442 (531h59m57.715791477s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844716407 (531h59m57.715799442s) [noise=true , address=45.91.116.8:4045 , piece_id=E4WILQH7BH76NF3KOO7QLTTQTJ5ZH4G53AAFEX54RYA23EFILKMA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844837908 (531h59m57.715920963s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844874618 (531h59m57.715957653s) [noise=true , address=45.91.116.14:4191 , piece_id=HT52BFE2QI2WY362KVTBI2TURA5QZBGOTUWCPFXXSSKTGJ57IYRA, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844970330 (531h59m57.716053375s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 27 19:28:57.844979608 (531h59m57.716062643s) [noise=true , address=45.91.116.11:4120 , piece_id=QRI4HZKB4XX7MTTDHBJ3AV2RXVTDSCAGGDV4P2RL7CQNTMVVBLVQ, seg_pos={0 0}, upload=466566]: Long tail cancellation due to optimal threshold Dec 27 19:28:57.844999486 (531h59m57.716082521s) [seg_pos={0 0}, upload=466566]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:28:57.845101981 (531h59m57.716185016s) [upload=466566]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:28:57.845120386 (531h59m57.716203421s) [upload=466566]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:28:58.136270631 (531h59m58.007353696s) [upload=466566]: Done uploading segments... Dec 27 19:28:58.136292552 (531h59m58.007375587s) [upload=466566]: Upload finished. err: <nil> Duration: 30.006058098s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.632484029s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:29:12 UTC) orbiter warning: context deadline exceeded Duration: 3.632523665s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.623302056s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:28:14 UTC) orbiter warning: context deadline exceeded Duration: 3.623332645s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......XX..XX..X.....X.........X....XXX..XX....X...XXXX...X..] duration: 6.179042813s trace: https://jaeger.dev.storj.io/trace/58259cc7a171b836 FAIL (2025-12-27 19:27:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9:25:27.963256488 (531h56m27.834339533s) [upload=466512]: Starting upload Dec 27 19:25:27.963328736 (531h56m27.834411781s) [upload=466512]: Done starting upload Dec 27 19:25:27.963385464 (531h56m27.834468509s) [upload=466512]: Uploading segments... Dec 27 19:25:27.963596546 (531h56m27.834679581s) [upload=466512]: Got next segment. Inline: false Dec 27 19:25:27.963644487 (531h56m27.834727532s) [upload=466512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:25:27.963650498 (531h56m27.834733533s) [upload=466512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:25:28.205689256 (531h56m28.076772311s) [seg_pos={0 0}, upload=466512]: Begin upload segment... Dec 27 19:25:28.205765160 (531h56m28.076848195s) [seg_pos={0 0}, upload=466512]: Done begin upload segment. Dec 27 19:25:28.205856323 (531h56m28.076939368s) [noise=true , address=145.79.227.16:4910 , piece_id=J7QGFTINHMFSCPXZPUGW62AH3D5AAVRM47V2BS77JRKZ6XD2G3YA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.205943790 (531h56m28.077026825s) [noise=true , address=145.79.227.7:4295 , piece_id=LW3OIONSIWXFPU5ZJPHRRTVHPWRXNB6VQOCVP6DKXNNUKQATOBBQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.207118517 (531h56m28.078201552s) [noise=true , address=145.79.227.52:4446 , piece_id=2WMNG6TMECEEUQOYAKGDYNLCIVVOAXM7IF3NU5IZHUKS6TB2RLLA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.207531213 (531h56m28.078614248s) [noise=true , address=45.91.116.13:4156 , piece_id=BOFGDLHVUGLVSD2GPAX7ANY3WTGFUSEJO7AR4JEN3DZ477RO723A, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.207971100 (531h56m28.079054135s) [noise=true , address=145.79.227.20:4438 , piece_id=6YBMCT4RYF4FEIBZ4KD3G3TSPC5CA2PEA4IX7V5KDHF6DHX3ZFTA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.208378236 (531h56m28.079461281s) [noise=true , address=145.79.227.10:4403 , piece_id=N7A76KDPIQCVJL53GZV2EW3O4PF5SCU6SHYTDGSODGT4PKUWGRNA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.208759802 (531h56m28.079842837s) [noise=true , address=77.90.31.11:4553 , piece_id=5EICALWXIYJT33WGUZVQMTSMPPC6ANKAIBLVKQXRIJJ2EM6UIYDQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.210974198 (531h56m28.082057233s) [noise=true , address=45.91.116.4:4939 , piece_id=KUA7AOQJBQ4GOE5OZ73INY44CIJBWG6MERD7I3CTO6H4A7VYYFMA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.211537372 (531h56m28.082620417s) [noise=true , address=145.79.227.12:4765 , piece_id=XXJYHZ4INIGRKG5DIKMYWT23EAI62N57QWILR37QPH2GF4O5AWWQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.212022215 (531h56m28.08310526s) [noise=true , address=145.79.227.45:4230 , piece_id=WGVSMNNM7ILJT2IP6YFQMJN4EGHGP3FHEFDNIT5MCQL5AEHYEFRQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.212574616 (531h56m28.083657661s) [noise=true , address=145.79.227.41:4087 , piece_id=7LJEYUQR23AWAOHE7DQRROX4TXUJD75I5ELPKDCQ5K75G2FCUEJA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.213431137 (531h56m28.084514182s) [noise=true , address=45.91.116.5:4298 , piece_id=H4ZKP7AXEY7TBKGMADGCL5HFF43SV5WKFRCJICLHXUKHMY4H653A, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.215697683 (531h56m28.086780718s) [noise=true , address=145.79.227.9:4365 , piece_id=BZIGMHKQ3QRUJBXL4G4354AZBN355CAOYY2FFTFQMHM6QQNW6I3Q, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.217633750 (531h56m28.088716805s) [noise=true , address=45.91.116.12:4225 , piece_id=L7RK2EWNXNEITJZBPKSJFOQV7ZTYF7QDKBPM2N37KINFMGXTXJZQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.219659307 (531h56m28.090742352s) [noise=true , address=77.90.31.15:4697 , piece_id=KOCUHT6WFRC4HG7JA55LVKRWRGP7WBVRZOTBW7ZMMREORK32DBDQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.221313918 (531h56m28.092396963s) [noise=true , address=145.79.227.38:4977 , piece_id=CSSC6O4QV2VXTKSFTDRF6N54HVXDKTAKBOF7QSX72ABQRKDF2P5A, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:29.896263456 (531h56m29.767346501s) [upload=466512]: Next returned nil segment Dec 27 19:25:29.896280739 (531h56m29.767363774s) [upload=466512]: Waiting for error group managing segments... Dec 27 19:25:30.215761130 (531h56m30.086844165s) [noise=true , address=145.79.227.9:4365 , piece_id=BZIGMHKQ3QRUJBXL4G4354AZBN355CAOYY2FFTFQMHM6QQNW6I3Q, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.219977318 (531h56m30.091060353s) [noise=true , address=145.79.227.20:4438 , piece_id=6YBMCT4RYF4FEIBZ4KD3G3TSPC5CA2PEA4IX7V5KDHF6DHX3ZFTA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.221632610 (531h56m30.092715645s) [noise=true , address=145.79.227.10:4403 , piece_id=N7A76KDPIQCVJL53GZV2EW3O4PF5SCU6SHYTDGSODGT4PKUWGRNA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.223305416 (531h56m30.094388461s) [noise=true , address=145.79.227.12:4765 , piece_id=XXJYHZ4INIGRKG5DIKMYWT23EAI62N57QWILR37QPH2GF4O5AWWQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.227030108 (531h56m30.098113153s) [noise=true , address=145.79.227.7:4295 , piece_id=LW3OIONSIWXFPU5ZJPHRRTVHPWRXNB6VQOCVP6DKXNNUKQATOBBQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.231888309 (531h56m30.102971344s) [noise=true , address=145.79.227.52:4446 , piece_id=2WMNG6TMECEEUQOYAKGDYNLCIVVOAXM7IF3NU5IZHUKS6TB2RLLA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.248341539 (531h56m30.119424574s) [noise=true , address=145.79.227.38:4977 , piece_id=CSSC6O4QV2VXTKSFTDRF6N54HVXDKTAKBOF7QSX72ABQRKDF2P5A, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.254031444 (531h56m30.125114479s) [noise=true , address=145.79.227.45:4230 , piece_id=WGVSMNNM7ILJT2IP6YFQMJN4EGHGP3FHEFDNIT5MCQL5AEHYEFRQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.257494618 (531h56m30.128577653s) [noise=true , address=145.79.227.41:4087 , piece_id=7LJEYUQR23AWAOHE7DQRROX4TXUJD75I5ELPKDCQ5K75G2FCUEJA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.393758039 (531h56m30.264841084s) [noise=true , address=77.90.31.11:4553 , piece_id=5EICALWXIYJT33WGUZVQMTSMPPC6ANKAIBLVKQXRIJJ2EM6UIYDQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.480480334 (531h56m30.351563369s) [noise=true , address=45.91.116.12:4225 , piece_id=L7RK2EWNXNEITJZBPKSJFOQV7ZTYF7QDKBPM2N37KINFMGXTXJZQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:31.159626217 (531h56m31.030709262s) [noise=true , address=77.90.31.15:4697 , piece_id=KOCUHT6WFRC4HG7JA55LVKRWRGP7WBVRZOTBW7ZMMREORK32DBDQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:31.195241895 (531h56m31.06632494s) [noise=true , address=45.91.116.4:4939 , piece_id=KUA7AOQJBQ4GOE5OZ73INY44CIJBWG6MERD7I3CTO6H4A7VYYFMA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:31.195263966 (531h56m31.066347011s) [seg_pos={0 0}, upload=466512]: Segment reached optimal threshold of 13 pieces. Dec 27 19:25:31.195473295 (531h56m31.06655633s) [noise=true , address=45.91.116.5:4298 , piece_id=H4ZKP7AXEY7TBKGMADGCL5HFF43SV5WKFRCJICLHXUKHMY4H653A, seg_pos={0 0}, upload=466512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 19:25:31.195480670 (531h56m31.066563715s) [noise=true , address=45.91.116.5:4298 , piece_id=H4ZKP7AXEY7TBKGMADGCL5HFF43SV5WKFRCJICLHXUKHMY4H653A, seg_pos={0 0}, upload=466512]: Long tail cancellation due to optimal threshold Dec 27 19:25:31.195638409 (531h56m31.066721445s) [noise=true , address=145.79.227.16:4910 , piece_id=J7QGFTINHMFSCPXZPUGW62AH3D5AAVRM47V2BS77JRKZ6XD2G3YA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52228->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52228->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52228->145.79.227.16:4910: use of closed network connection Dec 27 19:25:31.195646394 (531h56m31.06672944s) [noise=true , address=145.79.227.16:4910 , piece_id=J7QGFTINHMFSCPXZPUGW62AH3D5AAVRM47V2BS77JRKZ6XD2G3YA, seg_pos={0 0}, upload=466512]: Long tail cancellation due to optimal threshold Dec 27 19:25:31.195694146 (531h56m31.066777191s) [noise=true , address=45.91.116.13:4156 , piece_id=BOFGDLHVUGLVSD2GPAX7ANY3WTGFUSEJO7AR4JEN3DZ477RO723A, seg_pos={0 0}, upload=466512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 19:25:31.195715226 (531h56m31.066798271s) [noise=true , address=45.91.116.13:4156 , piece_id=BOFGDLHVUGLVSD2GPAX7ANY3WTGFUSEJO7AR4JEN3DZ477RO723A, seg_pos={0 0}, upload=466512]: Long tail cancellation due to optimal threshold Dec 27 19:25:31.195844913 (531h56m31.066927948s) [seg_pos={0 0}, upload=466512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:25:31.195993385 (531h56m31.06707643s) [upload=466512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:25:31.196018964 (531h56m31.067102009s) [upload=466512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:25:31.481357062 (531h56m31.352440117s) [upload=466512]: Done uploading segments... Dec 27 19:25:31.481382962 (531h56m31.352466007s) [upload=466512]: Upload finished. err: <nil> Duration: 6.179069295s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.53988255s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:27:07 UTC) orbiter warning: context deadline exceeded Duration: 3.539909391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.......XX..XX..X.....X.........X....XXX..XX....X...XXXX...X.] duration: 16.668056083s trace: https://jaeger.dev.storj.io/trace/58259cc7a171b836 FAIL (2025-12-27 19:26:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9:25:27.963256488 (531h56m27.834339533s) [upload=466512]: Starting upload Dec 27 19:25:27.963328736 (531h56m27.834411781s) [upload=466512]: Done starting upload Dec 27 19:25:27.963385464 (531h56m27.834468509s) [upload=466512]: Uploading segments... Dec 27 19:25:27.963596546 (531h56m27.834679581s) [upload=466512]: Got next segment. Inline: false Dec 27 19:25:27.963644487 (531h56m27.834727532s) [upload=466512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:25:27.963650498 (531h56m27.834733533s) [upload=466512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:25:28.205689256 (531h56m28.076772311s) [seg_pos={0 0}, upload=466512]: Begin upload segment... Dec 27 19:25:28.205765160 (531h56m28.076848195s) [seg_pos={0 0}, upload=466512]: Done begin upload segment. Dec 27 19:25:28.205856323 (531h56m28.076939368s) [noise=true , address=145.79.227.16:4910 , piece_id=J7QGFTINHMFSCPXZPUGW62AH3D5AAVRM47V2BS77JRKZ6XD2G3YA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.205943790 (531h56m28.077026825s) [noise=true , address=145.79.227.7:4295 , piece_id=LW3OIONSIWXFPU5ZJPHRRTVHPWRXNB6VQOCVP6DKXNNUKQATOBBQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.207118517 (531h56m28.078201552s) [noise=true , address=145.79.227.52:4446 , piece_id=2WMNG6TMECEEUQOYAKGDYNLCIVVOAXM7IF3NU5IZHUKS6TB2RLLA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.207531213 (531h56m28.078614248s) [noise=true , address=45.91.116.13:4156 , piece_id=BOFGDLHVUGLVSD2GPAX7ANY3WTGFUSEJO7AR4JEN3DZ477RO723A, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.207971100 (531h56m28.079054135s) [noise=true , address=145.79.227.20:4438 , piece_id=6YBMCT4RYF4FEIBZ4KD3G3TSPC5CA2PEA4IX7V5KDHF6DHX3ZFTA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.208378236 (531h56m28.079461281s) [noise=true , address=145.79.227.10:4403 , piece_id=N7A76KDPIQCVJL53GZV2EW3O4PF5SCU6SHYTDGSODGT4PKUWGRNA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.208759802 (531h56m28.079842837s) [noise=true , address=77.90.31.11:4553 , piece_id=5EICALWXIYJT33WGUZVQMTSMPPC6ANKAIBLVKQXRIJJ2EM6UIYDQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.210974198 (531h56m28.082057233s) [noise=true , address=45.91.116.4:4939 , piece_id=KUA7AOQJBQ4GOE5OZ73INY44CIJBWG6MERD7I3CTO6H4A7VYYFMA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.211537372 (531h56m28.082620417s) [noise=true , address=145.79.227.12:4765 , piece_id=XXJYHZ4INIGRKG5DIKMYWT23EAI62N57QWILR37QPH2GF4O5AWWQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.212022215 (531h56m28.08310526s) [noise=true , address=145.79.227.45:4230 , piece_id=WGVSMNNM7ILJT2IP6YFQMJN4EGHGP3FHEFDNIT5MCQL5AEHYEFRQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.212574616 (531h56m28.083657661s) [noise=true , address=145.79.227.41:4087 , piece_id=7LJEYUQR23AWAOHE7DQRROX4TXUJD75I5ELPKDCQ5K75G2FCUEJA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.213431137 (531h56m28.084514182s) [noise=true , address=45.91.116.5:4298 , piece_id=H4ZKP7AXEY7TBKGMADGCL5HFF43SV5WKFRCJICLHXUKHMY4H653A, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.215697683 (531h56m28.086780718s) [noise=true , address=145.79.227.9:4365 , piece_id=BZIGMHKQ3QRUJBXL4G4354AZBN355CAOYY2FFTFQMHM6QQNW6I3Q, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.217633750 (531h56m28.088716805s) [noise=true , address=45.91.116.12:4225 , piece_id=L7RK2EWNXNEITJZBPKSJFOQV7ZTYF7QDKBPM2N37KINFMGXTXJZQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.219659307 (531h56m28.090742352s) [noise=true , address=77.90.31.15:4697 , piece_id=KOCUHT6WFRC4HG7JA55LVKRWRGP7WBVRZOTBW7ZMMREORK32DBDQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.221313918 (531h56m28.092396963s) [noise=true , address=145.79.227.38:4977 , piece_id=CSSC6O4QV2VXTKSFTDRF6N54HVXDKTAKBOF7QSX72ABQRKDF2P5A, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:29.896263456 (531h56m29.767346501s) [upload=466512]: Next returned nil segment Dec 27 19:25:29.896280739 (531h56m29.767363774s) [upload=466512]: Waiting for error group managing segments... Dec 27 19:25:30.215761130 (531h56m30.086844165s) [noise=true , address=145.79.227.9:4365 , piece_id=BZIGMHKQ3QRUJBXL4G4354AZBN355CAOYY2FFTFQMHM6QQNW6I3Q, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.219977318 (531h56m30.091060353s) [noise=true , address=145.79.227.20:4438 , piece_id=6YBMCT4RYF4FEIBZ4KD3G3TSPC5CA2PEA4IX7V5KDHF6DHX3ZFTA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.221632610 (531h56m30.092715645s) [noise=true , address=145.79.227.10:4403 , piece_id=N7A76KDPIQCVJL53GZV2EW3O4PF5SCU6SHYTDGSODGT4PKUWGRNA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.223305416 (531h56m30.094388461s) [noise=true , address=145.79.227.12:4765 , piece_id=XXJYHZ4INIGRKG5DIKMYWT23EAI62N57QWILR37QPH2GF4O5AWWQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.227030108 (531h56m30.098113153s) [noise=true , address=145.79.227.7:4295 , piece_id=LW3OIONSIWXFPU5ZJPHRRTVHPWRXNB6VQOCVP6DKXNNUKQATOBBQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.231888309 (531h56m30.102971344s) [noise=true , address=145.79.227.52:4446 , piece_id=2WMNG6TMECEEUQOYAKGDYNLCIVVOAXM7IF3NU5IZHUKS6TB2RLLA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.248341539 (531h56m30.119424574s) [noise=true , address=145.79.227.38:4977 , piece_id=CSSC6O4QV2VXTKSFTDRF6N54HVXDKTAKBOF7QSX72ABQRKDF2P5A, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.254031444 (531h56m30.125114479s) [noise=true , address=145.79.227.45:4230 , piece_id=WGVSMNNM7ILJT2IP6YFQMJN4EGHGP3FHEFDNIT5MCQL5AEHYEFRQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.257494618 (531h56m30.128577653s) [noise=true , address=145.79.227.41:4087 , piece_id=7LJEYUQR23AWAOHE7DQRROX4TXUJD75I5ELPKDCQ5K75G2FCUEJA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.393758039 (531h56m30.264841084s) [noise=true , address=77.90.31.11:4553 , piece_id=5EICALWXIYJT33WGUZVQMTSMPPC6ANKAIBLVKQXRIJJ2EM6UIYDQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.480480334 (531h56m30.351563369s) [noise=true , address=45.91.116.12:4225 , piece_id=L7RK2EWNXNEITJZBPKSJFOQV7ZTYF7QDKBPM2N37KINFMGXTXJZQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:31.159626217 (531h56m31.030709262s) [noise=true , address=77.90.31.15:4697 , piece_id=KOCUHT6WFRC4HG7JA55LVKRWRGP7WBVRZOTBW7ZMMREORK32DBDQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:31.195241895 (531h56m31.06632494s) [noise=true , address=45.91.116.4:4939 , piece_id=KUA7AOQJBQ4GOE5OZ73INY44CIJBWG6MERD7I3CTO6H4A7VYYFMA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:31.195263966 (531h56m31.066347011s) [seg_pos={0 0}, upload=466512]: Segment reached optimal threshold of 13 pieces. Dec 27 19:25:31.195473295 (531h56m31.06655633s) [noise=true , address=45.91.116.5:4298 , piece_id=H4ZKP7AXEY7TBKGMADGCL5HFF43SV5WKFRCJICLHXUKHMY4H653A, seg_pos={0 0}, upload=466512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 19:25:31.195480670 (531h56m31.066563715s) [noise=true , address=45.91.116.5:4298 , piece_id=H4ZKP7AXEY7TBKGMADGCL5HFF43SV5WKFRCJICLHXUKHMY4H653A, seg_pos={0 0}, upload=466512]: Long tail cancellation due to optimal threshold Dec 27 19:25:31.195638409 (531h56m31.066721445s) [noise=true , address=145.79.227.16:4910 , piece_id=J7QGFTINHMFSCPXZPUGW62AH3D5AAVRM47V2BS77JRKZ6XD2G3YA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52228->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52228->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52228->145.79.227.16:4910: use of closed network connection Dec 27 19:25:31.195646394 (531h56m31.06672944s) [noise=true , address=145.79.227.16:4910 , piece_id=J7QGFTINHMFSCPXZPUGW62AH3D5AAVRM47V2BS77JRKZ6XD2G3YA, seg_pos={0 0}, upload=466512]: Long tail cancellation due to optimal threshold Dec 27 19:25:31.195694146 (531h56m31.066777191s) [noise=true , address=45.91.116.13:4156 , piece_id=BOFGDLHVUGLVSD2GPAX7ANY3WTGFUSEJO7AR4JEN3DZ477RO723A, seg_pos={0 0}, upload=466512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 19:25:31.195715226 (531h56m31.066798271s) [noise=true , address=45.91.116.13:4156 , piece_id=BOFGDLHVUGLVSD2GPAX7ANY3WTGFUSEJO7AR4JEN3DZ477RO723A, seg_pos={0 0}, upload=466512]: Long tail cancellation due to optimal threshold Dec 27 19:25:31.195844913 (531h56m31.066927948s) [seg_pos={0 0}, upload=466512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:25:31.195993385 (531h56m31.06707643s) [upload=466512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:25:31.196018964 (531h56m31.067102009s) [upload=466512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:25:31.481357062 (531h56m31.352440117s) [upload=466512]: Done uploading segments... Dec 27 19:25:31.481382962 (531h56m31.352466007s) [upload=466512]: Upload finished. err: <nil> Duration: 16.668096511s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.596490961s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:26:50 UTC) orbiter warning: context deadline exceeded Duration: 3.596516258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [........XX..XX..X.....X.........X....XXX..XX....X...XXXX...X] duration: 30.003487644s trace: https://jaeger.dev.storj.io/trace/58259cc7a171b836 FAIL (2025-12-27 19:25:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9:25:27.963256488 (531h56m27.834339533s) [upload=466512]: Starting upload Dec 27 19:25:27.963328736 (531h56m27.834411781s) [upload=466512]: Done starting upload Dec 27 19:25:27.963385464 (531h56m27.834468509s) [upload=466512]: Uploading segments... Dec 27 19:25:27.963596546 (531h56m27.834679581s) [upload=466512]: Got next segment. Inline: false Dec 27 19:25:27.963644487 (531h56m27.834727532s) [upload=466512]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:25:27.963650498 (531h56m27.834733533s) [upload=466512]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:25:28.205689256 (531h56m28.076772311s) [seg_pos={0 0}, upload=466512]: Begin upload segment... Dec 27 19:25:28.205765160 (531h56m28.076848195s) [seg_pos={0 0}, upload=466512]: Done begin upload segment. Dec 27 19:25:28.205856323 (531h56m28.076939368s) [noise=true , address=145.79.227.16:4910 , piece_id=J7QGFTINHMFSCPXZPUGW62AH3D5AAVRM47V2BS77JRKZ6XD2G3YA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.205943790 (531h56m28.077026825s) [noise=true , address=145.79.227.7:4295 , piece_id=LW3OIONSIWXFPU5ZJPHRRTVHPWRXNB6VQOCVP6DKXNNUKQATOBBQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.207118517 (531h56m28.078201552s) [noise=true , address=145.79.227.52:4446 , piece_id=2WMNG6TMECEEUQOYAKGDYNLCIVVOAXM7IF3NU5IZHUKS6TB2RLLA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.207531213 (531h56m28.078614248s) [noise=true , address=45.91.116.13:4156 , piece_id=BOFGDLHVUGLVSD2GPAX7ANY3WTGFUSEJO7AR4JEN3DZ477RO723A, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.207971100 (531h56m28.079054135s) [noise=true , address=145.79.227.20:4438 , piece_id=6YBMCT4RYF4FEIBZ4KD3G3TSPC5CA2PEA4IX7V5KDHF6DHX3ZFTA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.208378236 (531h56m28.079461281s) [noise=true , address=145.79.227.10:4403 , piece_id=N7A76KDPIQCVJL53GZV2EW3O4PF5SCU6SHYTDGSODGT4PKUWGRNA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.208759802 (531h56m28.079842837s) [noise=true , address=77.90.31.11:4553 , piece_id=5EICALWXIYJT33WGUZVQMTSMPPC6ANKAIBLVKQXRIJJ2EM6UIYDQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.210974198 (531h56m28.082057233s) [noise=true , address=45.91.116.4:4939 , piece_id=KUA7AOQJBQ4GOE5OZ73INY44CIJBWG6MERD7I3CTO6H4A7VYYFMA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.211537372 (531h56m28.082620417s) [noise=true , address=145.79.227.12:4765 , piece_id=XXJYHZ4INIGRKG5DIKMYWT23EAI62N57QWILR37QPH2GF4O5AWWQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.212022215 (531h56m28.08310526s) [noise=true , address=145.79.227.45:4230 , piece_id=WGVSMNNM7ILJT2IP6YFQMJN4EGHGP3FHEFDNIT5MCQL5AEHYEFRQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.212574616 (531h56m28.083657661s) [noise=true , address=145.79.227.41:4087 , piece_id=7LJEYUQR23AWAOHE7DQRROX4TXUJD75I5ELPKDCQ5K75G2FCUEJA, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.213431137 (531h56m28.084514182s) [noise=true , address=45.91.116.5:4298 , piece_id=H4ZKP7AXEY7TBKGMADGCL5HFF43SV5WKFRCJICLHXUKHMY4H653A, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.215697683 (531h56m28.086780718s) [noise=true , address=145.79.227.9:4365 , piece_id=BZIGMHKQ3QRUJBXL4G4354AZBN355CAOYY2FFTFQMHM6QQNW6I3Q, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.217633750 (531h56m28.088716805s) [noise=true , address=45.91.116.12:4225 , piece_id=L7RK2EWNXNEITJZBPKSJFOQV7ZTYF7QDKBPM2N37KINFMGXTXJZQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.219659307 (531h56m28.090742352s) [noise=true , address=77.90.31.15:4697 , piece_id=KOCUHT6WFRC4HG7JA55LVKRWRGP7WBVRZOTBW7ZMMREORK32DBDQ, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:28.221313918 (531h56m28.092396963s) [noise=true , address=145.79.227.38:4977 , piece_id=CSSC6O4QV2VXTKSFTDRF6N54HVXDKTAKBOF7QSX72ABQRKDF2P5A, seg_pos={0 0}, upload=466512]: Uploading piece... Dec 27 19:25:29.896263456 (531h56m29.767346501s) [upload=466512]: Next returned nil segment Dec 27 19:25:29.896280739 (531h56m29.767363774s) [upload=466512]: Waiting for error group managing segments... Dec 27 19:25:30.215761130 (531h56m30.086844165s) [noise=true , address=145.79.227.9:4365 , piece_id=BZIGMHKQ3QRUJBXL4G4354AZBN355CAOYY2FFTFQMHM6QQNW6I3Q, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.219977318 (531h56m30.091060353s) [noise=true , address=145.79.227.20:4438 , piece_id=6YBMCT4RYF4FEIBZ4KD3G3TSPC5CA2PEA4IX7V5KDHF6DHX3ZFTA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.221632610 (531h56m30.092715645s) [noise=true , address=145.79.227.10:4403 , piece_id=N7A76KDPIQCVJL53GZV2EW3O4PF5SCU6SHYTDGSODGT4PKUWGRNA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.223305416 (531h56m30.094388461s) [noise=true , address=145.79.227.12:4765 , piece_id=XXJYHZ4INIGRKG5DIKMYWT23EAI62N57QWILR37QPH2GF4O5AWWQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.227030108 (531h56m30.098113153s) [noise=true , address=145.79.227.7:4295 , piece_id=LW3OIONSIWXFPU5ZJPHRRTVHPWRXNB6VQOCVP6DKXNNUKQATOBBQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.231888309 (531h56m30.102971344s) [noise=true , address=145.79.227.52:4446 , piece_id=2WMNG6TMECEEUQOYAKGDYNLCIVVOAXM7IF3NU5IZHUKS6TB2RLLA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.248341539 (531h56m30.119424574s) [noise=true , address=145.79.227.38:4977 , piece_id=CSSC6O4QV2VXTKSFTDRF6N54HVXDKTAKBOF7QSX72ABQRKDF2P5A, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.254031444 (531h56m30.125114479s) [noise=true , address=145.79.227.45:4230 , piece_id=WGVSMNNM7ILJT2IP6YFQMJN4EGHGP3FHEFDNIT5MCQL5AEHYEFRQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.257494618 (531h56m30.128577653s) [noise=true , address=145.79.227.41:4087 , piece_id=7LJEYUQR23AWAOHE7DQRROX4TXUJD75I5ELPKDCQ5K75G2FCUEJA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.393758039 (531h56m30.264841084s) [noise=true , address=77.90.31.11:4553 , piece_id=5EICALWXIYJT33WGUZVQMTSMPPC6ANKAIBLVKQXRIJJ2EM6UIYDQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:30.480480334 (531h56m30.351563369s) [noise=true , address=45.91.116.12:4225 , piece_id=L7RK2EWNXNEITJZBPKSJFOQV7ZTYF7QDKBPM2N37KINFMGXTXJZQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:31.159626217 (531h56m31.030709262s) [noise=true , address=77.90.31.15:4697 , piece_id=KOCUHT6WFRC4HG7JA55LVKRWRGP7WBVRZOTBW7ZMMREORK32DBDQ, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:31.195241895 (531h56m31.06632494s) [noise=true , address=45.91.116.4:4939 , piece_id=KUA7AOQJBQ4GOE5OZ73INY44CIJBWG6MERD7I3CTO6H4A7VYYFMA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: <nil> Dec 27 19:25:31.195263966 (531h56m31.066347011s) [seg_pos={0 0}, upload=466512]: Segment reached optimal threshold of 13 pieces. Dec 27 19:25:31.195473295 (531h56m31.06655633s) [noise=true , address=45.91.116.5:4298 , piece_id=H4ZKP7AXEY7TBKGMADGCL5HFF43SV5WKFRCJICLHXUKHMY4H653A, seg_pos={0 0}, upload=466512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 19:25:31.195480670 (531h56m31.066563715s) [noise=true , address=45.91.116.5:4298 , piece_id=H4ZKP7AXEY7TBKGMADGCL5HFF43SV5WKFRCJICLHXUKHMY4H653A, seg_pos={0 0}, upload=466512]: Long tail cancellation due to optimal threshold Dec 27 19:25:31.195638409 (531h56m31.066721445s) [noise=true , address=145.79.227.16:4910 , piece_id=J7QGFTINHMFSCPXZPUGW62AH3D5AAVRM47V2BS77JRKZ6XD2G3YA, seg_pos={0 0}, upload=466512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:52228->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:52228->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52228->145.79.227.16:4910: use of closed network connection Dec 27 19:25:31.195646394 (531h56m31.06672944s) [noise=true , address=145.79.227.16:4910 , piece_id=J7QGFTINHMFSCPXZPUGW62AH3D5AAVRM47V2BS77JRKZ6XD2G3YA, seg_pos={0 0}, upload=466512]: Long tail cancellation due to optimal threshold Dec 27 19:25:31.195694146 (531h56m31.066777191s) [noise=true , address=45.91.116.13:4156 , piece_id=BOFGDLHVUGLVSD2GPAX7ANY3WTGFUSEJO7AR4JEN3DZ477RO723A, seg_pos={0 0}, upload=466512]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 19:25:31.195715226 (531h56m31.066798271s) [noise=true , address=45.91.116.13:4156 , piece_id=BOFGDLHVUGLVSD2GPAX7ANY3WTGFUSEJO7AR4JEN3DZ477RO723A, seg_pos={0 0}, upload=466512]: Long tail cancellation due to optimal threshold Dec 27 19:25:31.195844913 (531h56m31.066927948s) [seg_pos={0 0}, upload=466512]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:25:31.195993385 (531h56m31.06707643s) [upload=466512]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:25:31.196018964 (531h56m31.067102009s) [upload=466512]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:25:31.481357062 (531h56m31.352440117s) [upload=466512]: Done uploading segments... Dec 27 19:25:31.481382962 (531h56m31.352466007s) [upload=466512]: Upload finished. err: <nil> Duration: 30.003529603s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.618409159s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:25:38 UTC) orbiter warning: context deadline exceeded Duration: 3.618438364s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.541679562s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:25:14 UTC) orbiter warning: context deadline exceeded Duration: 3.541702174s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X........XX..XX..X.....X.........X....XXX..XX....X...XXXX...] duration: 8.115295033s trace: https://jaeger.dev.storj.io/trace/648b342c73d29d5 FAIL (2025-12-27 19:24:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .. Dec 27 19:23:16.501591743 (531h54m16.372674778s) [upload=466476]: Got next segment. Inline: false Dec 27 19:23:16.501634725 (531h54m16.37271779s) [upload=466476]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:23:16.501641207 (531h54m16.372724262s) [upload=466476]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:23:16.754880984 (531h54m16.625964029s) [seg_pos={0 0}, upload=466476]: Begin upload segment... Dec 27 19:23:16.754935818 (531h54m16.626018853s) [seg_pos={0 0}, upload=466476]: Done begin upload segment. Dec 27 19:23:16.754970564 (531h54m16.626053599s) [noise=true , address=145.79.227.14:4839 , piece_id=AQMFEEYQAD54QJWPQX6ZP3WTBGAHS2WCXTBT6MQN2PJD4NA25HSQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755000672 (531h54m16.626083717s) [noise=true , address=145.79.227.13:4803 , piece_id=KCN4O6WYRN7F6GCVGU3KO6QVYITENVDONEDLXAYKIZSN6L4GJFNQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755503099 (531h54m16.626586144s) [noise=true , address=145.79.227.3:4149 , piece_id=H3W46S6TQLJYZ44HI4IBNGZRUYXP3UEHCTDH2ECONB55W763KOFA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755903170 (531h54m16.626986205s) [noise=true , address=45.91.116.10:4082 , piece_id=FG3RP5V7HH2FPCTD6NQQ3BTSAHZCI6RDZSTVZXOHQJ267HPJHD3A, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.756640464 (531h54m16.627723499s) [noise=true , address=77.90.31.11:4554 , piece_id=4V6OCERUKYG55QNZE5ZILJVEEPNHYYABFVQDCKY6EMHSS5KZSVTA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.757072898 (531h54m16.628155933s) [noise=true , address=77.90.31.15:4697 , piece_id=DZASZCJXJQKMCGAUBFCGD2BJB6MVA4F27BCMOSGUCGYAFLMI77SA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.757488079 (531h54m16.628571114s) [noise=true , address=45.91.116.8:4047 , piece_id=4RUWLO6GEVLF5VICEOJCLGWZPG7RJZORUOJHNJYOYYFMZZRAC2FQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.759021448 (531h54m16.630104514s) [noise=true , address=145.79.227.4:4185 , piece_id=VOHIPHF6VSYGLQ36J2PSSXF2XH5LTXDBJTY6BX6M7MWO2FSKOFJQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.759168348 (531h54m16.630251393s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.760324640 (531h54m16.631407675s) [noise=true , address=145.79.227.32:4834 , piece_id=ETBLHLWPB2JALLE7CN3SUYLASCLUVR26ZGIQEISYOMQY733IHHXA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.761729965 (531h54m16.632813s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.763168495 (531h54m16.63425154s) [noise=true , address=145.79.227.48:4302 , piece_id=TQ3CGUA6VDPGUKRN5KANSAZBNUPS4IB577ERH3ZXJFPXGTNWWGUQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.764701254 (531h54m16.635784299s) [noise=true , address=145.79.227.25:4618 , piece_id=L3PXL2MIDVJQNKD3S6U7RTCR4BYSZASEZX4BFQTHSNWJDVEHWIJQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.766267606 (531h54m16.637350641s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.767775116 (531h54m16.638858152s) [noise=true , address=45.91.116.13:4156 , piece_id=R3ZW5PXGTA2HE7ULC4OL6MTCY6LCUNQIDPWKWSN47A76X7MVD7KA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.768870272 (531h54m16.639953307s) [noise=true , address=145.79.227.51:4411 , piece_id=RO3IOHK3PHGXUDULCJFBPEWA4BA5FL4NMVBJRSEV3EADFZIIMY3A, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:18.483523994 (531h54m18.354607039s) [upload=466476]: Next returned nil segment Dec 27 19:23:18.483537440 (531h54m18.354620475s) [upload=466476]: Waiting for error group managing segments... Dec 27 19:23:18.762597582 (531h54m18.633680627s) [noise=true , address=145.79.227.14:4839 , piece_id=AQMFEEYQAD54QJWPQX6ZP3WTBGAHS2WCXTBT6MQN2PJD4NA25HSQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.768595143 (531h54m18.639678188s) [noise=true , address=145.79.227.25:4618 , piece_id=L3PXL2MIDVJQNKD3S6U7RTCR4BYSZASEZX4BFQTHSNWJDVEHWIJQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.771732026 (531h54m18.642815061s) [noise=true , address=145.79.227.51:4411 , piece_id=RO3IOHK3PHGXUDULCJFBPEWA4BA5FL4NMVBJRSEV3EADFZIIMY3A, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.773080293 (531h54m18.644163338s) [noise=true , address=145.79.227.4:4185 , piece_id=VOHIPHF6VSYGLQ36J2PSSXF2XH5LTXDBJTY6BX6M7MWO2FSKOFJQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.775605912 (531h54m18.646688947s) [noise=true , address=145.79.227.13:4803 , piece_id=KCN4O6WYRN7F6GCVGU3KO6QVYITENVDONEDLXAYKIZSN6L4GJFNQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.785048393 (531h54m18.656131428s) [noise=true , address=145.79.227.48:4302 , piece_id=TQ3CGUA6VDPGUKRN5KANSAZBNUPS4IB577ERH3ZXJFPXGTNWWGUQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.785901868 (531h54m18.656984903s) [noise=true , address=145.79.227.3:4149 , piece_id=H3W46S6TQLJYZ44HI4IBNGZRUYXP3UEHCTDH2ECONB55W763KOFA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.793918794 (531h54m18.665001829s) [noise=true , address=145.79.227.32:4834 , piece_id=ETBLHLWPB2JALLE7CN3SUYLASCLUVR26ZGIQEISYOMQY733IHHXA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.938862404 (531h54m18.809945439s) [noise=true , address=77.90.31.11:4554 , piece_id=4V6OCERUKYG55QNZE5ZILJVEEPNHYYABFVQDCKY6EMHSS5KZSVTA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.999147918 (531h54m18.870230973s) [noise=true , address=45.91.116.8:4047 , piece_id=4RUWLO6GEVLF5VICEOJCLGWZPG7RJZORUOJHNJYOYYFMZZRAC2FQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.011417874 (531h54m18.882500919s) [noise=true , address=77.90.31.15:4697 , piece_id=DZASZCJXJQKMCGAUBFCGD2BJB6MVA4F27BCMOSGUCGYAFLMI77SA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.026965258 (531h54m18.898048293s) [noise=true , address=45.91.116.13:4156 , piece_id=R3ZW5PXGTA2HE7ULC4OL6MTCY6LCUNQIDPWKWSN47A76X7MVD7KA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.161276372 (531h54m19.032359408s) [noise=true , address=45.91.116.10:4082 , piece_id=FG3RP5V7HH2FPCTD6NQQ3BTSAHZCI6RDZSTVZXOHQJ267HPJHD3A, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.161295669 (531h54m19.032378704s) [seg_pos={0 0}, upload=466476]: Segment reached optimal threshold of 13 pieces. Dec 27 19:23:19.161475221 (531h54m19.032558256s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 19:23:19.161481583 (531h54m19.032564628s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161574129 (531h54m19.032657174s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection Dec 27 19:23:19.161583728 (531h54m19.032666763s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161841568 (531h54m19.032924644s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection Dec 27 19:23:19.161852138 (531h54m19.032935173s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161883859 (531h54m19.032966904s) [seg_pos={0 0}, upload=466476]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:23:19.161970434 (531h54m19.033053469s) [upload=466476]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:23:19.161978569 (531h54m19.033061604s) [upload=466476]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:23:19.449828641 (531h54m19.320911676s) [upload=466476]: Done uploading segments... Dec 27 19:23:19.449847937 (531h54m19.320930982s) [upload=466476]: Upload finished. err: <nil> Duration: 8.115311113s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.549105022s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:24:36 UTC) orbiter warning: context deadline exceeded Duration: 3.54913126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX........XX..XX..X.....X.........X....XXX..XX....X...XXXX..] duration: 5.96143109s trace: https://jaeger.dev.storj.io/trace/648b342c73d29d5 FAIL (2025-12-27 19:24:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .. Dec 27 19:23:16.501591743 (531h54m16.372674778s) [upload=466476]: Got next segment. Inline: false Dec 27 19:23:16.501634725 (531h54m16.37271779s) [upload=466476]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:23:16.501641207 (531h54m16.372724262s) [upload=466476]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:23:16.754880984 (531h54m16.625964029s) [seg_pos={0 0}, upload=466476]: Begin upload segment... Dec 27 19:23:16.754935818 (531h54m16.626018853s) [seg_pos={0 0}, upload=466476]: Done begin upload segment. Dec 27 19:23:16.754970564 (531h54m16.626053599s) [noise=true , address=145.79.227.14:4839 , piece_id=AQMFEEYQAD54QJWPQX6ZP3WTBGAHS2WCXTBT6MQN2PJD4NA25HSQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755000672 (531h54m16.626083717s) [noise=true , address=145.79.227.13:4803 , piece_id=KCN4O6WYRN7F6GCVGU3KO6QVYITENVDONEDLXAYKIZSN6L4GJFNQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755503099 (531h54m16.626586144s) [noise=true , address=145.79.227.3:4149 , piece_id=H3W46S6TQLJYZ44HI4IBNGZRUYXP3UEHCTDH2ECONB55W763KOFA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755903170 (531h54m16.626986205s) [noise=true , address=45.91.116.10:4082 , piece_id=FG3RP5V7HH2FPCTD6NQQ3BTSAHZCI6RDZSTVZXOHQJ267HPJHD3A, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.756640464 (531h54m16.627723499s) [noise=true , address=77.90.31.11:4554 , piece_id=4V6OCERUKYG55QNZE5ZILJVEEPNHYYABFVQDCKY6EMHSS5KZSVTA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.757072898 (531h54m16.628155933s) [noise=true , address=77.90.31.15:4697 , piece_id=DZASZCJXJQKMCGAUBFCGD2BJB6MVA4F27BCMOSGUCGYAFLMI77SA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.757488079 (531h54m16.628571114s) [noise=true , address=45.91.116.8:4047 , piece_id=4RUWLO6GEVLF5VICEOJCLGWZPG7RJZORUOJHNJYOYYFMZZRAC2FQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.759021448 (531h54m16.630104514s) [noise=true , address=145.79.227.4:4185 , piece_id=VOHIPHF6VSYGLQ36J2PSSXF2XH5LTXDBJTY6BX6M7MWO2FSKOFJQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.759168348 (531h54m16.630251393s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.760324640 (531h54m16.631407675s) [noise=true , address=145.79.227.32:4834 , piece_id=ETBLHLWPB2JALLE7CN3SUYLASCLUVR26ZGIQEISYOMQY733IHHXA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.761729965 (531h54m16.632813s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.763168495 (531h54m16.63425154s) [noise=true , address=145.79.227.48:4302 , piece_id=TQ3CGUA6VDPGUKRN5KANSAZBNUPS4IB577ERH3ZXJFPXGTNWWGUQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.764701254 (531h54m16.635784299s) [noise=true , address=145.79.227.25:4618 , piece_id=L3PXL2MIDVJQNKD3S6U7RTCR4BYSZASEZX4BFQTHSNWJDVEHWIJQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.766267606 (531h54m16.637350641s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.767775116 (531h54m16.638858152s) [noise=true , address=45.91.116.13:4156 , piece_id=R3ZW5PXGTA2HE7ULC4OL6MTCY6LCUNQIDPWKWSN47A76X7MVD7KA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.768870272 (531h54m16.639953307s) [noise=true , address=145.79.227.51:4411 , piece_id=RO3IOHK3PHGXUDULCJFBPEWA4BA5FL4NMVBJRSEV3EADFZIIMY3A, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:18.483523994 (531h54m18.354607039s) [upload=466476]: Next returned nil segment Dec 27 19:23:18.483537440 (531h54m18.354620475s) [upload=466476]: Waiting for error group managing segments... Dec 27 19:23:18.762597582 (531h54m18.633680627s) [noise=true , address=145.79.227.14:4839 , piece_id=AQMFEEYQAD54QJWPQX6ZP3WTBGAHS2WCXTBT6MQN2PJD4NA25HSQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.768595143 (531h54m18.639678188s) [noise=true , address=145.79.227.25:4618 , piece_id=L3PXL2MIDVJQNKD3S6U7RTCR4BYSZASEZX4BFQTHSNWJDVEHWIJQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.771732026 (531h54m18.642815061s) [noise=true , address=145.79.227.51:4411 , piece_id=RO3IOHK3PHGXUDULCJFBPEWA4BA5FL4NMVBJRSEV3EADFZIIMY3A, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.773080293 (531h54m18.644163338s) [noise=true , address=145.79.227.4:4185 , piece_id=VOHIPHF6VSYGLQ36J2PSSXF2XH5LTXDBJTY6BX6M7MWO2FSKOFJQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.775605912 (531h54m18.646688947s) [noise=true , address=145.79.227.13:4803 , piece_id=KCN4O6WYRN7F6GCVGU3KO6QVYITENVDONEDLXAYKIZSN6L4GJFNQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.785048393 (531h54m18.656131428s) [noise=true , address=145.79.227.48:4302 , piece_id=TQ3CGUA6VDPGUKRN5KANSAZBNUPS4IB577ERH3ZXJFPXGTNWWGUQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.785901868 (531h54m18.656984903s) [noise=true , address=145.79.227.3:4149 , piece_id=H3W46S6TQLJYZ44HI4IBNGZRUYXP3UEHCTDH2ECONB55W763KOFA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.793918794 (531h54m18.665001829s) [noise=true , address=145.79.227.32:4834 , piece_id=ETBLHLWPB2JALLE7CN3SUYLASCLUVR26ZGIQEISYOMQY733IHHXA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.938862404 (531h54m18.809945439s) [noise=true , address=77.90.31.11:4554 , piece_id=4V6OCERUKYG55QNZE5ZILJVEEPNHYYABFVQDCKY6EMHSS5KZSVTA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.999147918 (531h54m18.870230973s) [noise=true , address=45.91.116.8:4047 , piece_id=4RUWLO6GEVLF5VICEOJCLGWZPG7RJZORUOJHNJYOYYFMZZRAC2FQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.011417874 (531h54m18.882500919s) [noise=true , address=77.90.31.15:4697 , piece_id=DZASZCJXJQKMCGAUBFCGD2BJB6MVA4F27BCMOSGUCGYAFLMI77SA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.026965258 (531h54m18.898048293s) [noise=true , address=45.91.116.13:4156 , piece_id=R3ZW5PXGTA2HE7ULC4OL6MTCY6LCUNQIDPWKWSN47A76X7MVD7KA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.161276372 (531h54m19.032359408s) [noise=true , address=45.91.116.10:4082 , piece_id=FG3RP5V7HH2FPCTD6NQQ3BTSAHZCI6RDZSTVZXOHQJ267HPJHD3A, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.161295669 (531h54m19.032378704s) [seg_pos={0 0}, upload=466476]: Segment reached optimal threshold of 13 pieces. Dec 27 19:23:19.161475221 (531h54m19.032558256s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 19:23:19.161481583 (531h54m19.032564628s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161574129 (531h54m19.032657174s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection Dec 27 19:23:19.161583728 (531h54m19.032666763s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161841568 (531h54m19.032924644s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection Dec 27 19:23:19.161852138 (531h54m19.032935173s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161883859 (531h54m19.032966904s) [seg_pos={0 0}, upload=466476]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:23:19.161970434 (531h54m19.033053469s) [upload=466476]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:23:19.161978569 (531h54m19.033061604s) [upload=466476]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:23:19.449828641 (531h54m19.320911676s) [upload=466476]: Done uploading segments... Dec 27 19:23:19.449847937 (531h54m19.320930982s) [upload=466476]: Upload finished. err: <nil> Duration: 5.961464255s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.549293459s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:24:07 UTC) orbiter warning: context deadline exceeded Duration: 3.549322705s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX........XX..XX..X.....X.........X....XXX..XX....X...XXXX.] duration: 5.910627184s trace: https://jaeger.dev.storj.io/trace/648b342c73d29d5 FAIL (2025-12-27 19:23:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .. Dec 27 19:23:16.501591743 (531h54m16.372674778s) [upload=466476]: Got next segment. Inline: false Dec 27 19:23:16.501634725 (531h54m16.37271779s) [upload=466476]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:23:16.501641207 (531h54m16.372724262s) [upload=466476]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:23:16.754880984 (531h54m16.625964029s) [seg_pos={0 0}, upload=466476]: Begin upload segment... Dec 27 19:23:16.754935818 (531h54m16.626018853s) [seg_pos={0 0}, upload=466476]: Done begin upload segment. Dec 27 19:23:16.754970564 (531h54m16.626053599s) [noise=true , address=145.79.227.14:4839 , piece_id=AQMFEEYQAD54QJWPQX6ZP3WTBGAHS2WCXTBT6MQN2PJD4NA25HSQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755000672 (531h54m16.626083717s) [noise=true , address=145.79.227.13:4803 , piece_id=KCN4O6WYRN7F6GCVGU3KO6QVYITENVDONEDLXAYKIZSN6L4GJFNQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755503099 (531h54m16.626586144s) [noise=true , address=145.79.227.3:4149 , piece_id=H3W46S6TQLJYZ44HI4IBNGZRUYXP3UEHCTDH2ECONB55W763KOFA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755903170 (531h54m16.626986205s) [noise=true , address=45.91.116.10:4082 , piece_id=FG3RP5V7HH2FPCTD6NQQ3BTSAHZCI6RDZSTVZXOHQJ267HPJHD3A, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.756640464 (531h54m16.627723499s) [noise=true , address=77.90.31.11:4554 , piece_id=4V6OCERUKYG55QNZE5ZILJVEEPNHYYABFVQDCKY6EMHSS5KZSVTA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.757072898 (531h54m16.628155933s) [noise=true , address=77.90.31.15:4697 , piece_id=DZASZCJXJQKMCGAUBFCGD2BJB6MVA4F27BCMOSGUCGYAFLMI77SA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.757488079 (531h54m16.628571114s) [noise=true , address=45.91.116.8:4047 , piece_id=4RUWLO6GEVLF5VICEOJCLGWZPG7RJZORUOJHNJYOYYFMZZRAC2FQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.759021448 (531h54m16.630104514s) [noise=true , address=145.79.227.4:4185 , piece_id=VOHIPHF6VSYGLQ36J2PSSXF2XH5LTXDBJTY6BX6M7MWO2FSKOFJQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.759168348 (531h54m16.630251393s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.760324640 (531h54m16.631407675s) [noise=true , address=145.79.227.32:4834 , piece_id=ETBLHLWPB2JALLE7CN3SUYLASCLUVR26ZGIQEISYOMQY733IHHXA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.761729965 (531h54m16.632813s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.763168495 (531h54m16.63425154s) [noise=true , address=145.79.227.48:4302 , piece_id=TQ3CGUA6VDPGUKRN5KANSAZBNUPS4IB577ERH3ZXJFPXGTNWWGUQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.764701254 (531h54m16.635784299s) [noise=true , address=145.79.227.25:4618 , piece_id=L3PXL2MIDVJQNKD3S6U7RTCR4BYSZASEZX4BFQTHSNWJDVEHWIJQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.766267606 (531h54m16.637350641s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.767775116 (531h54m16.638858152s) [noise=true , address=45.91.116.13:4156 , piece_id=R3ZW5PXGTA2HE7ULC4OL6MTCY6LCUNQIDPWKWSN47A76X7MVD7KA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.768870272 (531h54m16.639953307s) [noise=true , address=145.79.227.51:4411 , piece_id=RO3IOHK3PHGXUDULCJFBPEWA4BA5FL4NMVBJRSEV3EADFZIIMY3A, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:18.483523994 (531h54m18.354607039s) [upload=466476]: Next returned nil segment Dec 27 19:23:18.483537440 (531h54m18.354620475s) [upload=466476]: Waiting for error group managing segments... Dec 27 19:23:18.762597582 (531h54m18.633680627s) [noise=true , address=145.79.227.14:4839 , piece_id=AQMFEEYQAD54QJWPQX6ZP3WTBGAHS2WCXTBT6MQN2PJD4NA25HSQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.768595143 (531h54m18.639678188s) [noise=true , address=145.79.227.25:4618 , piece_id=L3PXL2MIDVJQNKD3S6U7RTCR4BYSZASEZX4BFQTHSNWJDVEHWIJQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.771732026 (531h54m18.642815061s) [noise=true , address=145.79.227.51:4411 , piece_id=RO3IOHK3PHGXUDULCJFBPEWA4BA5FL4NMVBJRSEV3EADFZIIMY3A, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.773080293 (531h54m18.644163338s) [noise=true , address=145.79.227.4:4185 , piece_id=VOHIPHF6VSYGLQ36J2PSSXF2XH5LTXDBJTY6BX6M7MWO2FSKOFJQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.775605912 (531h54m18.646688947s) [noise=true , address=145.79.227.13:4803 , piece_id=KCN4O6WYRN7F6GCVGU3KO6QVYITENVDONEDLXAYKIZSN6L4GJFNQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.785048393 (531h54m18.656131428s) [noise=true , address=145.79.227.48:4302 , piece_id=TQ3CGUA6VDPGUKRN5KANSAZBNUPS4IB577ERH3ZXJFPXGTNWWGUQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.785901868 (531h54m18.656984903s) [noise=true , address=145.79.227.3:4149 , piece_id=H3W46S6TQLJYZ44HI4IBNGZRUYXP3UEHCTDH2ECONB55W763KOFA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.793918794 (531h54m18.665001829s) [noise=true , address=145.79.227.32:4834 , piece_id=ETBLHLWPB2JALLE7CN3SUYLASCLUVR26ZGIQEISYOMQY733IHHXA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.938862404 (531h54m18.809945439s) [noise=true , address=77.90.31.11:4554 , piece_id=4V6OCERUKYG55QNZE5ZILJVEEPNHYYABFVQDCKY6EMHSS5KZSVTA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.999147918 (531h54m18.870230973s) [noise=true , address=45.91.116.8:4047 , piece_id=4RUWLO6GEVLF5VICEOJCLGWZPG7RJZORUOJHNJYOYYFMZZRAC2FQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.011417874 (531h54m18.882500919s) [noise=true , address=77.90.31.15:4697 , piece_id=DZASZCJXJQKMCGAUBFCGD2BJB6MVA4F27BCMOSGUCGYAFLMI77SA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.026965258 (531h54m18.898048293s) [noise=true , address=45.91.116.13:4156 , piece_id=R3ZW5PXGTA2HE7ULC4OL6MTCY6LCUNQIDPWKWSN47A76X7MVD7KA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.161276372 (531h54m19.032359408s) [noise=true , address=45.91.116.10:4082 , piece_id=FG3RP5V7HH2FPCTD6NQQ3BTSAHZCI6RDZSTVZXOHQJ267HPJHD3A, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.161295669 (531h54m19.032378704s) [seg_pos={0 0}, upload=466476]: Segment reached optimal threshold of 13 pieces. Dec 27 19:23:19.161475221 (531h54m19.032558256s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 19:23:19.161481583 (531h54m19.032564628s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161574129 (531h54m19.032657174s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection Dec 27 19:23:19.161583728 (531h54m19.032666763s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161841568 (531h54m19.032924644s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection Dec 27 19:23:19.161852138 (531h54m19.032935173s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161883859 (531h54m19.032966904s) [seg_pos={0 0}, upload=466476]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:23:19.161970434 (531h54m19.033053469s) [upload=466476]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:23:19.161978569 (531h54m19.033061604s) [upload=466476]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:23:19.449828641 (531h54m19.320911676s) [upload=466476]: Done uploading segments... Dec 27 19:23:19.449847937 (531h54m19.320930982s) [upload=466476]: Upload finished. err: <nil> Duration: 5.910644937s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXX........XX..XX..X.....X.........X....XXX..XX....X...XXXX] duration: 30.004340394s trace: https://jaeger.dev.storj.io/trace/648b342c73d29d5 FAIL (2025-12-27 19:23:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== .. Dec 27 19:23:16.501591743 (531h54m16.372674778s) [upload=466476]: Got next segment. Inline: false Dec 27 19:23:16.501634725 (531h54m16.37271779s) [upload=466476]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:23:16.501641207 (531h54m16.372724262s) [upload=466476]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:23:16.754880984 (531h54m16.625964029s) [seg_pos={0 0}, upload=466476]: Begin upload segment... Dec 27 19:23:16.754935818 (531h54m16.626018853s) [seg_pos={0 0}, upload=466476]: Done begin upload segment. Dec 27 19:23:16.754970564 (531h54m16.626053599s) [noise=true , address=145.79.227.14:4839 , piece_id=AQMFEEYQAD54QJWPQX6ZP3WTBGAHS2WCXTBT6MQN2PJD4NA25HSQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755000672 (531h54m16.626083717s) [noise=true , address=145.79.227.13:4803 , piece_id=KCN4O6WYRN7F6GCVGU3KO6QVYITENVDONEDLXAYKIZSN6L4GJFNQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755503099 (531h54m16.626586144s) [noise=true , address=145.79.227.3:4149 , piece_id=H3W46S6TQLJYZ44HI4IBNGZRUYXP3UEHCTDH2ECONB55W763KOFA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.755903170 (531h54m16.626986205s) [noise=true , address=45.91.116.10:4082 , piece_id=FG3RP5V7HH2FPCTD6NQQ3BTSAHZCI6RDZSTVZXOHQJ267HPJHD3A, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.756640464 (531h54m16.627723499s) [noise=true , address=77.90.31.11:4554 , piece_id=4V6OCERUKYG55QNZE5ZILJVEEPNHYYABFVQDCKY6EMHSS5KZSVTA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.757072898 (531h54m16.628155933s) [noise=true , address=77.90.31.15:4697 , piece_id=DZASZCJXJQKMCGAUBFCGD2BJB6MVA4F27BCMOSGUCGYAFLMI77SA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.757488079 (531h54m16.628571114s) [noise=true , address=45.91.116.8:4047 , piece_id=4RUWLO6GEVLF5VICEOJCLGWZPG7RJZORUOJHNJYOYYFMZZRAC2FQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.759021448 (531h54m16.630104514s) [noise=true , address=145.79.227.4:4185 , piece_id=VOHIPHF6VSYGLQ36J2PSSXF2XH5LTXDBJTY6BX6M7MWO2FSKOFJQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.759168348 (531h54m16.630251393s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.760324640 (531h54m16.631407675s) [noise=true , address=145.79.227.32:4834 , piece_id=ETBLHLWPB2JALLE7CN3SUYLASCLUVR26ZGIQEISYOMQY733IHHXA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.761729965 (531h54m16.632813s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.763168495 (531h54m16.63425154s) [noise=true , address=145.79.227.48:4302 , piece_id=TQ3CGUA6VDPGUKRN5KANSAZBNUPS4IB577ERH3ZXJFPXGTNWWGUQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.764701254 (531h54m16.635784299s) [noise=true , address=145.79.227.25:4618 , piece_id=L3PXL2MIDVJQNKD3S6U7RTCR4BYSZASEZX4BFQTHSNWJDVEHWIJQ, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.766267606 (531h54m16.637350641s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.767775116 (531h54m16.638858152s) [noise=true , address=45.91.116.13:4156 , piece_id=R3ZW5PXGTA2HE7ULC4OL6MTCY6LCUNQIDPWKWSN47A76X7MVD7KA, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:16.768870272 (531h54m16.639953307s) [noise=true , address=145.79.227.51:4411 , piece_id=RO3IOHK3PHGXUDULCJFBPEWA4BA5FL4NMVBJRSEV3EADFZIIMY3A, seg_pos={0 0}, upload=466476]: Uploading piece... Dec 27 19:23:18.483523994 (531h54m18.354607039s) [upload=466476]: Next returned nil segment Dec 27 19:23:18.483537440 (531h54m18.354620475s) [upload=466476]: Waiting for error group managing segments... Dec 27 19:23:18.762597582 (531h54m18.633680627s) [noise=true , address=145.79.227.14:4839 , piece_id=AQMFEEYQAD54QJWPQX6ZP3WTBGAHS2WCXTBT6MQN2PJD4NA25HSQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.768595143 (531h54m18.639678188s) [noise=true , address=145.79.227.25:4618 , piece_id=L3PXL2MIDVJQNKD3S6U7RTCR4BYSZASEZX4BFQTHSNWJDVEHWIJQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.771732026 (531h54m18.642815061s) [noise=true , address=145.79.227.51:4411 , piece_id=RO3IOHK3PHGXUDULCJFBPEWA4BA5FL4NMVBJRSEV3EADFZIIMY3A, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.773080293 (531h54m18.644163338s) [noise=true , address=145.79.227.4:4185 , piece_id=VOHIPHF6VSYGLQ36J2PSSXF2XH5LTXDBJTY6BX6M7MWO2FSKOFJQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.775605912 (531h54m18.646688947s) [noise=true , address=145.79.227.13:4803 , piece_id=KCN4O6WYRN7F6GCVGU3KO6QVYITENVDONEDLXAYKIZSN6L4GJFNQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.785048393 (531h54m18.656131428s) [noise=true , address=145.79.227.48:4302 , piece_id=TQ3CGUA6VDPGUKRN5KANSAZBNUPS4IB577ERH3ZXJFPXGTNWWGUQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.785901868 (531h54m18.656984903s) [noise=true , address=145.79.227.3:4149 , piece_id=H3W46S6TQLJYZ44HI4IBNGZRUYXP3UEHCTDH2ECONB55W763KOFA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.793918794 (531h54m18.665001829s) [noise=true , address=145.79.227.32:4834 , piece_id=ETBLHLWPB2JALLE7CN3SUYLASCLUVR26ZGIQEISYOMQY733IHHXA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.938862404 (531h54m18.809945439s) [noise=true , address=77.90.31.11:4554 , piece_id=4V6OCERUKYG55QNZE5ZILJVEEPNHYYABFVQDCKY6EMHSS5KZSVTA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:18.999147918 (531h54m18.870230973s) [noise=true , address=45.91.116.8:4047 , piece_id=4RUWLO6GEVLF5VICEOJCLGWZPG7RJZORUOJHNJYOYYFMZZRAC2FQ, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.011417874 (531h54m18.882500919s) [noise=true , address=77.90.31.15:4697 , piece_id=DZASZCJXJQKMCGAUBFCGD2BJB6MVA4F27BCMOSGUCGYAFLMI77SA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.026965258 (531h54m18.898048293s) [noise=true , address=45.91.116.13:4156 , piece_id=R3ZW5PXGTA2HE7ULC4OL6MTCY6LCUNQIDPWKWSN47A76X7MVD7KA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.161276372 (531h54m19.032359408s) [noise=true , address=45.91.116.10:4082 , piece_id=FG3RP5V7HH2FPCTD6NQQ3BTSAHZCI6RDZSTVZXOHQJ267HPJHD3A, seg_pos={0 0}, upload=466476]: Done uploading piece. err: <nil> Dec 27 19:23:19.161295669 (531h54m19.032378704s) [seg_pos={0 0}, upload=466476]: Segment reached optimal threshold of 13 pieces. Dec 27 19:23:19.161475221 (531h54m19.032558256s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 19:23:19.161481583 (531h54m19.032564628s) [noise=true , address=45.91.116.12:4226 , piece_id=RT6TTCMX6Z54D2LHHPMFAEOWTJR46XCLOETFKBB7KDVN35FSSCBA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161574129 (531h54m19.032657174s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE): protocol: write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection; write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46952->145.79.227.5:4223: use of closed network connection Dec 27 19:23:19.161583728 (531h54m19.032666763s) [noise=true , address=145.79.227.5:4223 , piece_id=RIZ5RSV6EEACAFNGRUBH5G7TWP7ORILXC5RG4TFTY6AE6HG2VAVA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161841568 (531h54m19.032924644s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57646->145.79.227.16:4909: use of closed network connection Dec 27 19:23:19.161852138 (531h54m19.032935173s) [noise=true , address=145.79.227.16:4909 , piece_id=3R5COVXWCKS5ARBA27CF7372MMFDETJZMNC6GPMZPYPFVP5VJZCA, seg_pos={0 0}, upload=466476]: Long tail cancellation due to optimal threshold Dec 27 19:23:19.161883859 (531h54m19.032966904s) [seg_pos={0 0}, upload=466476]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:23:19.161970434 (531h54m19.033053469s) [upload=466476]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:23:19.161978569 (531h54m19.033061604s) [upload=466476]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:23:19.449828641 (531h54m19.320911676s) [upload=466476]: Done uploading segments... Dec 27 19:23:19.449847937 (531h54m19.320930982s) [upload=466476]: Upload finished. err: <nil> Duration: 30.004363687s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.567493336s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:23:38 UTC) orbiter warning: context deadline exceeded Duration: 3.567510989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.577145829s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:23:22 UTC) orbiter warning: context deadline exceeded Duration: 3.577184713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.522231618s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:22:56 UTC) orbiter warning: context deadline exceeded Duration: 3.522258529s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XXXX........XX..XX..X.....X.........X....XXX..XX....X...XXX] duration: 30.003953888s trace: https://jaeger.dev.storj.io/trace/63f2f1644d73313c FAIL (2025-12-27 19:22:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: UEDPOKX6AK7HAARANWIP56M2MVOWGOU5IRE34DXDWMK67TMZE7IQ) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P5GGZWHMDPHBQ6S4D42BYW7FZRAOCPT55XUXRDIGAH5JW2PCTMMQ) context deadline exceeded ======LOGS====== Dec 27 19:22:10.960205057 (531h53m10.831288092s) [upload=466462]: Starting upload Dec 27 19:22:10.960259301 (531h53m10.831342336s) [upload=466462]: Done starting upload Dec 27 19:22:10.960391463 (531h53m10.831474508s) [upload=466462]: Uploading segments... Dec 27 19:22:10.960483237 (531h53m10.831566272s) [upload=466462]: Got next segment. Inline: false Dec 27 19:22:10.960515368 (531h53m10.831598403s) [upload=466462]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:22:10.960521570 (531h53m10.831604605s) [upload=466462]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:22:11.207630532 (531h53m11.078713577s) [seg_pos={0 0}, upload=466462]: Begin upload segment... Dec 27 19:22:11.207719110 (531h53m11.078802155s) [seg_pos={0 0}, upload=466462]: Done begin upload segment. Dec 27 19:22:11.207765869 (531h53m11.078848914s) [noise=true , address=45.91.116.10:4084 , piece_id=RPDR6VBP2MTVN5TP76NMNAOPW3QER5U4Z5JOQPXHGGYHFK6WOX5A, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.207791147 (531h53m11.078874182s) [noise=true , address=145.79.227.39:4015 , piece_id=3RUKGNKIPPZKJKUJRFZWE6SFUFHCBQVAZ32556DRJLN5ZJSS6DRQ, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.208422139 (531h53m11.079505184s) [noise=true , address=77.90.31.11:4554 , piece_id=GR5GEL3IGHYOBNJQH6SXJPRMMTP27JN3LXAQ5P5AIRVAWE5EOCJA, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.208830807 (531h53m11.079913852s) [noise=true , address=77.90.31.13:4629 , piece_id=E2WRNSLROSLGANXWE4ENP7P3KB4UGYNOE7IGIBG3FYMGWPJ2ONVA, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.209288659 (531h53m11.080371704s) [noise=true , address=45.91.116.5:4298 , piece_id=P5GGZWHMDPHBQ6S4D42BYW7FZRAOCPT55XUXRDIGAH5JW2PCTMMQ, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.209304149 (531h53m11.080387194s) [noise=true , address=77.90.31.10:4517 , piece_id=QOEY5U463HJKDIBVNACVGG5JU2XW6WQOYN4Y5YVZ553FQAPQAEDA, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.209614009 (531h53m11.080697044s) [noise=true , address=145.79.227.46:4603 , piece_id=YCTENQAWUASCINW74THZM3CAPDD4AWFD3DHVXZIEY5EBL3IUEZOQ, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.209769985 (531h53m11.080853031s) [noise=true , address=145.79.227.18:4007 , piece_id=LYVY2I5LVM3Q4T3GQ6ZZ6WLLRLQJ74EEXSTXLLMEWL4YUIMYN4HA, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.210385317 (531h53m11.081468353s) [noise=true , address=145.79.227.51:4410 , piece_id=N3ACBRJOBY5B4NJDABDCNEEMFXUSPGRDERAO4XWU4GD46WCPOHLQ, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.210562695 (531h53m11.08164574s) [noise=true , address=145.79.227.28:4727 , piece_id=L7RDACSYPEFQTJCIY2YIM3ALQFNITVCVJKL7ELKQKTI2ZQXGBHKQ, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.211038852 (531h53m11.082121897s) [noise=true , address=145.79.227.4:4187 , piece_id=GYUT3O72KFCUGVUEIH2F3QQAYW2YBZ4LRHVO5K4RQKM7LFG37NUQ, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.211359823 (531h53m11.082442868s) [noise=true , address=145.79.227.16:4910 , piece_id=C2RO4DRRSJCVZVZVLRNMPI4DA6MQUW67NSBNM6YFDYOBWPI3NSKA, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.211806804 (531h53m11.082889849s) [noise=true , address=145.79.227.8:4329 , piece_id=JLBGAWAPKPSWVOXGEQSLSOJZAREHQALFARJPSAINXMMLDUW426SQ, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.212151309 (531h53m11.083234345s) [noise=true , address=45.91.116.9:4264 , piece_id=UEDPOKX6AK7HAARANWIP56M2MVOWGOU5IRE34DXDWMK67TMZE7IQ, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.212531974 (531h53m11.083615009s) [noise=true , address=77.90.31.18:4805 , piece_id=TLDLOM3SUW4FULQ7OF3WSARQV47U4OEV32MC36EVVCTRCLED57CA, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:11.212901388 (531h53m11.083984423s) [noise=true , address=145.79.227.41:4085 , piece_id=DBJKWALTV3FUFITGTABQOZ5YK7TC3AG2MYGD3YXACPCIXGUYWHXA, seg_pos={0 0}, upload=466462]: Uploading piece... Dec 27 19:22:12.822005333 (531h53m12.693088368s) [upload=466462]: Next returned nil segment Dec 27 19:22:12.822019891 (531h53m12.693102926s) [upload=466462]: Waiting for error group managing segments... Dec 27 19:22:13.204337825 (531h53m13.07542086s) [noise=true , address=145.79.227.4:4187 , piece_id=GYUT3O72KFCUGVUEIH2F3QQAYW2YBZ4LRHVO5K4RQKM7LFG37NUQ, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.211271349 (531h53m13.082354384s) [noise=true , address=145.79.227.51:4410 , piece_id=N3ACBRJOBY5B4NJDABDCNEEMFXUSPGRDERAO4XWU4GD46WCPOHLQ, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.216251822 (531h53m13.087334868s) [noise=true , address=145.79.227.18:4007 , piece_id=LYVY2I5LVM3Q4T3GQ6ZZ6WLLRLQJ74EEXSTXLLMEWL4YUIMYN4HA, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.219452297 (531h53m13.090535342s) [noise=true , address=145.79.227.46:4603 , piece_id=YCTENQAWUASCINW74THZM3CAPDD4AWFD3DHVXZIEY5EBL3IUEZOQ, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.224378907 (531h53m13.095461942s) [noise=true , address=145.79.227.28:4727 , piece_id=L7RDACSYPEFQTJCIY2YIM3ALQFNITVCVJKL7ELKQKTI2ZQXGBHKQ, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.224697754 (531h53m13.095780789s) [noise=true , address=145.79.227.39:4015 , piece_id=3RUKGNKIPPZKJKUJRFZWE6SFUFHCBQVAZ32556DRJLN5ZJSS6DRQ, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.232772690 (531h53m13.103855736s) [noise=true , address=145.79.227.41:4085 , piece_id=DBJKWALTV3FUFITGTABQOZ5YK7TC3AG2MYGD3YXACPCIXGUYWHXA, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.242558676 (531h53m13.113641721s) [noise=true , address=145.79.227.8:4329 , piece_id=JLBGAWAPKPSWVOXGEQSLSOJZAREHQALFARJPSAINXMMLDUW426SQ, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.373382358 (531h53m13.244465403s) [noise=true , address=145.79.227.16:4910 , piece_id=C2RO4DRRSJCVZVZVLRNMPI4DA6MQUW67NSBNM6YFDYOBWPI3NSKA, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.448586551 (531h53m13.319669586s) [noise=true , address=45.91.116.9:4264 , piece_id=UEDPOKX6AK7HAARANWIP56M2MVOWGOU5IRE34DXDWMK67TMZE7IQ, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.465298905 (531h53m13.33638194s) [noise=true , address=45.91.116.5:4298 , piece_id=P5GGZWHMDPHBQ6S4D42BYW7FZRAOCPT55XUXRDIGAH5JW2PCTMMQ, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:13.486940194 (531h53m13.358023239s) [noise=true , address=77.90.31.11:4554 , piece_id=GR5GEL3IGHYOBNJQH6SXJPRMMTP27JN3LXAQ5P5AIRVAWE5EOCJA, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:14.055457888 (531h53m13.926540933s) [noise=true , address=77.90.31.10:4517 , piece_id=QOEY5U463HJKDIBVNACVGG5JU2XW6WQOYN4Y5YVZ553FQAPQAEDA, seg_pos={0 0}, upload=466462]: Done uploading piece. err: <nil> Dec 27 19:22:14.055481112 (531h53m13.926564147s) [seg_pos={0 0}, upload=466462]: Segment reached optimal threshold of 13 pieces. Dec 27 19:22:14.055694859 (531h53m13.926777904s) [noise=true , address=77.90.31.13:4629 , piece_id=E2WRNSLROSLGANXWE4ENP7P3KB4UGYNOE7IGIBG3FYMGWPJ2ONVA, seg_pos={0 0}, upload=466462]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 27 19:22:14.055736609 (531h53m13.926819664s) [noise=true , address=77.90.31.13:4629 , piece_id=E2WRNSLROSLGANXWE4ENP7P3KB4UGYNOE7IGIBG3FYMGWPJ2ONVA, seg_pos={0 0}, upload=466462]: Long tail cancellation due to optimal threshold Dec 27 19:22:14.055883568 (531h53m13.926966613s) [noise=true , address=77.90.31.18:4805 , piece_id=TLDLOM3SUW4FULQ7OF3WSARQV47U4OEV32MC36EVVCTRCLED57CA, seg_pos={0 0}, upload=466462]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 19:22:14.055893607 (531h53m13.926976652s) [noise=true , address=77.90.31.18:4805 , piece_id=TLDLOM3SUW4FULQ7OF3WSARQV47U4OEV32MC36EVVCTRCLED57CA, seg_pos={0 0}, upload=466462]: Long tail cancellation due to optimal threshold Dec 27 19:22:14.056012364 (531h53m13.927095399s) [noise=true , address=45.91.116.10:4084 , piece_id=RPDR6VBP2MTVN5TP76NMNAOPW3QER5U4Z5JOQPXHGGYHFK6WOX5A, seg_pos={0 0}, upload=466462]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 27 19:22:14.056041790 (531h53m13.927124835s) [noise=true , address=45.91.116.10:4084 , piece_id=RPDR6VBP2MTVN5TP76NMNAOPW3QER5U4Z5JOQPXHGGYHFK6WOX5A, seg_pos={0 0}, upload=466462]: Long tail cancellation due to optimal threshold Dec 27 19:22:14.056078780 (531h53m13.927161815s) [seg_pos={0 0}, upload=466462]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:22:14.056185343 (531h53m13.927268378s) [upload=466462]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:22:14.056206473 (531h53m13.927289508s) [upload=466462]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:22:14.345017543 (531h53m14.216100589s) [upload=466462]: Done uploading segments... Dec 27 19:22:14.345036971 (531h53m14.216120006s) [upload=466462]: Upload finished. err: <nil> Duration: 30.0039871s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.586005575s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:22:25 UTC) orbiter warning: context deadline exceeded Duration: 3.586030393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.574947721s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:22:03 UTC) orbiter warning: context deadline exceeded Duration: 3.57496863s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XXXX........XX..XX..X.....X.........X....XXX..XX....X...XX] duration: 30.003242134s trace: https://jaeger.dev.storj.io/trace/38716a397713f617 FAIL (2025-12-27 19:21:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DL24TZJLSRYUH6E7GJ4BXV7IMR4HVAVG3SE2B7XZM3DPWJRGKSNQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: TQI6MLILNXE5PJE5L6CUSPKGJVFCKEIYV72SA5NTFJOWVXMANOOA) context deadline exceeded ======LOGS====== Dec 27 19:20:59.062521522 (531h51m58.933604568s) [upload=466446]: Starting upload Dec 27 19:20:59.062583451 (531h51m58.933666486s) [upload=466446]: Done starting upload Dec 27 19:20:59.062837043 (531h51m58.933920078s) [upload=466446]: Uploading segments... Dec 27 19:20:59.062952323 (531h51m58.934035368s) [upload=466446]: Got next segment. Inline: false Dec 27 19:20:59.063009912 (531h51m58.934092947s) [upload=466446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:20:59.063016556 (531h51m58.934099601s) [upload=466446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:20:59.312424705 (531h51m59.18350774s) [seg_pos={0 0}, upload=466446]: Begin upload segment... Dec 27 19:20:59.312471765 (531h51m59.1835548s) [seg_pos={0 0}, upload=466446]: Done begin upload segment. Dec 27 19:20:59.312497053 (531h51m59.183580088s) [noise=true , address=145.79.227.38:4977 , piece_id=5EYCG23DGR7L3QPAGFKLYDRNKLHFCCBYPD2SCLCGM67VFNGSCZYA, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.313345579 (531h51m59.184428624s) [noise=true , address=145.79.227.44:4194 , piece_id=6L5KZGJS3SSFXFLOCJWREK5W3TVWQWWIQX4XAIKINWQB7SM6K6AA, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.313886799 (531h51m59.184969834s) [noise=true , address=145.79.227.26:4653 , piece_id=Y4POQ7EELD5PJHSWKD4CI7GKK3GW6CR5Z3QWNZ4AS363BYD4ZKPQ, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.314264148 (531h51m59.185347193s) [noise=true , address=145.79.227.31:4799 , piece_id=BSZWBODQ54NHZQUJYLONMV2TZKQCB6PXKILSNJZVFF6ZGREEDJXQ, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.314572666 (531h51m59.185655701s) [noise=true , address=45.91.116.5:4300 , piece_id=HX4TPOFVP6E77CSX2XYJDXUA7NVK5GZTAFYLJSMJGFIGMLRMVR5A, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.314913605 (531h51m59.18599665s) [noise=true , address=45.91.116.6:4975 , piece_id=IAYZRSWLLBTO4WP24UYVDOQ36UWOSD5K3ZFQXPE2RVBUTG4YH6QA, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.315311512 (531h51m59.186394547s) [noise=true , address=145.79.227.10:4402 , piece_id=4D22Z6REUYS2KJ6B2ODB5W7K27YZNRU7G3NPUGON6FK6B274JKYA, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.316596800 (531h51m59.187679835s) [noise=true , address=145.79.227.37:4565 , piece_id=S532CFRJZQPJYFU4QRNLFWAGGIJWPIZL655MY5URY3BORAATCIAQ, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.317861428 (531h51m59.188944463s) [noise=true , address=45.91.116.10:4081 , piece_id=DL24TZJLSRYUH6E7GJ4BXV7IMR4HVAVG3SE2B7XZM3DPWJRGKSNQ, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.319165491 (531h51m59.190248526s) [noise=true , address=45.91.116.14:4192 , piece_id=MM3CFF24TDQ6KR7NT7KPIZCBZO32FEXANU3NIWZ37JNI35V3T5LQ, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.320508668 (531h51m59.191591703s) [noise=true , address=145.79.227.30:4762 , piece_id=2I2YG2TBYYWNKJ2ZFIYRO6FNV5L5VCNN7ADYE3T74OZ6IVB4BOFA, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.321551614 (531h51m59.192634649s) [noise=true , address=145.79.227.16:4909 , piece_id=FQAFMK27BOPUKKSNMXI7AA2XSAOQHPJNPB7SEIRWBIJLTF72HSSA, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.322619488 (531h51m59.193702523s) [noise=true , address=145.79.227.4:4185 , piece_id=ZCKGXBIEKKUKRR2EEE25JKSW3BDG2TNUATUBLJEKYSAUDOEUN3EQ, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.323201547 (531h51m59.194284582s) [noise=true , address=145.79.227.20:4438 , piece_id=AVIGUGALWNE5XXKDZ6PEE5SYI5JUIM4ZLNPEKSBVIQQICJ4ADW4A, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.324215908 (531h51m59.195298943s) [noise=true , address=145.79.227.2:4113 , piece_id=FNBPUVXMGNLF6QHQPMKM7B4WIBVSFIWI5N2YT6RQYLF7GTOH7BSA, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:20:59.324388867 (531h51m59.195471903s) [noise=true , address=77.90.31.19:4844 , piece_id=TQI6MLILNXE5PJE5L6CUSPKGJVFCKEIYV72SA5NTFJOWVXMANOOA, seg_pos={0 0}, upload=466446]: Uploading piece... Dec 27 19:21:01.031812804 (531h52m0.902895849s) [upload=466446]: Next returned nil segment Dec 27 19:21:01.031829887 (531h52m0.902912922s) [upload=466446]: Waiting for error group managing segments... Dec 27 19:21:01.317311277 (531h52m1.188394322s) [noise=true , address=145.79.227.37:4565 , piece_id=S532CFRJZQPJYFU4QRNLFWAGGIJWPIZL655MY5URY3BORAATCIAQ, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:01.321781238 (531h52m1.192864283s) [noise=true , address=145.79.227.31:4799 , piece_id=BSZWBODQ54NHZQUJYLONMV2TZKQCB6PXKILSNJZVFF6ZGREEDJXQ, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:01.325349593 (531h52m1.196432638s) [noise=true , address=145.79.227.2:4113 , piece_id=FNBPUVXMGNLF6QHQPMKM7B4WIBVSFIWI5N2YT6RQYLF7GTOH7BSA, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:01.328624399 (531h52m1.199707444s) [noise=true , address=145.79.227.38:4977 , piece_id=5EYCG23DGR7L3QPAGFKLYDRNKLHFCCBYPD2SCLCGM67VFNGSCZYA, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:01.330623024 (531h52m1.201706079s) [noise=true , address=145.79.227.26:4653 , piece_id=Y4POQ7EELD5PJHSWKD4CI7GKK3GW6CR5Z3QWNZ4AS363BYD4ZKPQ, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:01.335148471 (531h52m1.206231526s) [noise=true , address=145.79.227.16:4909 , piece_id=FQAFMK27BOPUKKSNMXI7AA2XSAOQHPJNPB7SEIRWBIJLTF72HSSA, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:01.337547369 (531h52m1.208630414s) [noise=true , address=145.79.227.10:4402 , piece_id=4D22Z6REUYS2KJ6B2ODB5W7K27YZNRU7G3NPUGON6FK6B274JKYA, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:01.347433355 (531h52m1.2185164s) [noise=true , address=145.79.227.4:4185 , piece_id=ZCKGXBIEKKUKRR2EEE25JKSW3BDG2TNUATUBLJEKYSAUDOEUN3EQ, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:01.366120569 (531h52m1.237203604s) [noise=true , address=145.79.227.20:4438 , piece_id=AVIGUGALWNE5XXKDZ6PEE5SYI5JUIM4ZLNPEKSBVIQQICJ4ADW4A, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:01.532090916 (531h52m1.403173961s) [noise=true , address=45.91.116.14:4192 , piece_id=MM3CFF24TDQ6KR7NT7KPIZCBZO32FEXANU3NIWZ37JNI35V3T5LQ, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:01.533423775 (531h52m1.40450682s) [noise=true , address=77.90.31.19:4844 , piece_id=TQI6MLILNXE5PJE5L6CUSPKGJVFCKEIYV72SA5NTFJOWVXMANOOA, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:02.017166010 (531h52m1.888249055s) [noise=true , address=45.91.116.10:4081 , piece_id=DL24TZJLSRYUH6E7GJ4BXV7IMR4HVAVG3SE2B7XZM3DPWJRGKSNQ, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:02.314458351 (531h52m2.185541396s) [noise=true , address=45.91.116.6:4975 , piece_id=IAYZRSWLLBTO4WP24UYVDOQ36UWOSD5K3ZFQXPE2RVBUTG4YH6QA, seg_pos={0 0}, upload=466446]: Done uploading piece. err: <nil> Dec 27 19:21:02.314483438 (531h52m2.185566483s) [seg_pos={0 0}, upload=466446]: Segment reached optimal threshold of 13 pieces. Dec 27 19:21:02.314713096 (531h52m2.185796131s) [noise=true , address=145.79.227.44:4194 , piece_id=6L5KZGJS3SSFXFLOCJWREK5W3TVWQWWIQX4XAIKINWQB7SM6K6AA, seg_pos={0 0}, upload=466446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 27 19:21:02.314720269 (531h52m2.185803304s) [noise=true , address=145.79.227.44:4194 , piece_id=6L5KZGJS3SSFXFLOCJWREK5W3TVWQWWIQX4XAIKINWQB7SM6K6AA, seg_pos={0 0}, upload=466446]: Long tail cancellation due to optimal threshold Dec 27 19:21:02.314901334 (531h52m2.185984379s) [noise=true , address=45.91.116.5:4300 , piece_id=HX4TPOFVP6E77CSX2XYJDXUA7NVK5GZTAFYLJSMJGFIGMLRMVR5A, seg_pos={0 0}, upload=466446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 19:21:02.314927323 (531h52m2.186010368s) [noise=true , address=45.91.116.5:4300 , piece_id=HX4TPOFVP6E77CSX2XYJDXUA7NVK5GZTAFYLJSMJGFIGMLRMVR5A, seg_pos={0 0}, upload=466446]: Long tail cancellation due to optimal threshold Dec 27 19:21:02.314980946 (531h52m2.186063981s) [noise=true , address=145.79.227.30:4762 , piece_id=2I2YG2TBYYWNKJ2ZFIYRO6FNV5L5VCNN7ADYE3T74OZ6IVB4BOFA, seg_pos={0 0}, upload=466446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 19:21:02.314988530 (531h52m2.186071575s) [noise=true , address=145.79.227.30:4762 , piece_id=2I2YG2TBYYWNKJ2ZFIYRO6FNV5L5VCNN7ADYE3T74OZ6IVB4BOFA, seg_pos={0 0}, upload=466446]: Long tail cancellation due to optimal threshold Dec 27 19:21:02.315038836 (531h52m2.186121881s) [seg_pos={0 0}, upload=466446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:21:02.315124820 (531h52m2.186207865s) [upload=466446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:21:02.315129949 (531h52m2.186212984s) [upload=466446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:21:02.614487572 (531h52m2.485570617s) [upload=466446]: Done uploading segments... Dec 27 19:21:02.614529602 (531h52m2.485612647s) [upload=466446]: Upload finished. err: <nil> Duration: 30.003265027s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.491787457s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:20:50 UTC) orbiter warning: context deadline exceeded Duration: 3.491830881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX.XXXX........XX..XX..X.....X.........X....XXX..XX....X...X] duration: 30.002395149s trace: https://jaeger.dev.storj.io/trace/70aa3de64e853af6 FAIL (2025-12-27 19:20:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 19:20:19.744558759 (531h51m19.615641805s) [upload=466438]: Starting upload Dec 27 19:20:19.744645014 (531h51m19.615728059s) [upload=466438]: Done starting upload Dec 27 19:20:19.744817613 (531h51m19.615900648s) [upload=466438]: Uploading segments... Dec 27 19:20:19.745199731 (531h51m19.616282776s) [upload=466438]: Got next segment. Inline: false Dec 27 19:20:19.745251819 (531h51m19.616334854s) [upload=466438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:20:19.745258522 (531h51m19.616341577s) [upload=466438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:20:19.992670681 (531h51m19.863753716s) [seg_pos={0 0}, upload=466438]: Begin upload segment... Dec 27 19:20:19.992722379 (531h51m19.863805424s) [seg_pos={0 0}, upload=466438]: Done begin upload segment. Dec 27 19:20:19.992748759 (531h51m19.863831794s) [noise=true , address=45.91.116.5:4299 , piece_id=GHSCRDWXOMDQLKT4AA7ILJCLPYIHRM6E6Y7CRJDYYBVRKEDXMIPQ, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.992843210 (531h51m19.863926255s) [noise=true , address=145.79.227.4:4187 , piece_id=KLUHIEALQRM6W36VGJSFRK6YDWU4VC22MIPYFVBAETWR3C6G27JQ, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.994031993 (531h51m19.865115028s) [noise=true , address=45.91.116.13:4153 , piece_id=MQXGNMYEARVR5UQXGHLXEEL5EMQFB2RRBZJNJJLYGP4CR6BTOUOQ, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.994413319 (531h51m19.865496354s) [noise=true , address=45.91.116.12:4227 , piece_id=NXCIPXKWLPDSQYEDSEKHN2FGRNSFNO3CA2F3RTEGP32BRD3R6FMQ, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.994868887 (531h51m19.865951922s) [noise=true , address=77.90.31.21:4878 , piece_id=CTNIAP5SXN7XTHAH5VL2B3XSED7BCHBRQFYYHHNAWN4HDWOEVBSA, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.995194116 (531h51m19.866277151s) [noise=true , address=145.79.227.7:4295 , piece_id=MBN4USSWEM7B5TQQFABJWR2U6ZOODV3ABUTACIMJGRG72GZ3WMSQ, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.995538271 (531h51m19.866621306s) [noise=true , address=145.79.227.17:4972 , piece_id=YA7BKMUB7CD7ZQNWSL5UHSNILEFNE6LGC3XGKMT2XIDNIBT4QF3Q, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.995881124 (531h51m19.866964169s) [noise=true , address=145.79.227.28:4725 , piece_id=SLHZ3AOMVXBF2HIMKCLBB2DRVGVBMWL3PS6TRRAPXD63JMUG6OIQ, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.996215841 (531h51m19.867298886s) [noise=true , address=145.79.227.50:4374 , piece_id=ZN5RNMMAD7PHGD54DA4CU2RXBKCWFV2ULVO5Q6GEGYLY7FWJ7ESA, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.996611094 (531h51m19.867694129s) [noise=true , address=145.79.227.31:4799 , piece_id=4AENG77AW7IVIEUIMMADTWA3PA45TA6OKAJAWKARWIRV2PT7LQ3Q, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.996972933 (531h51m19.868055978s) [noise=true , address=145.79.227.39:4013 , piece_id=3UAAOUCBBVAJPDXYQP4DVSVZYGA5OMRJ5GKTYP2LNJ4ZBBBJG7HQ, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.998146668 (531h51m19.869229703s) [noise=true , address=45.91.116.6:4975 , piece_id=X26CKZUKRFIWYBXVFXRNLWU6C6PBS4NVVNF67OEQX75AVWOKBYHQ, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.998767360 (531h51m19.869850395s) [noise=true , address=145.79.227.42:4122 , piece_id=YHRGPUUAXUVH6TYL6OOKFCZ7UCL5LBYGQYYMY2J3LUBE3L7RZ6GQ, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.999101435 (531h51m19.870184481s) [noise=true , address=145.79.227.15:4874 , piece_id=NQTZF44NBHSZNSUKLXW7FVOVXJ7OHDRL577WXHZ2YUVOIOGZC46Q, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.999409282 (531h51m19.870492317s) [noise=true , address=77.90.31.10:4518 , piece_id=O3LVLMFDUDB7CKWFHCHM5FQALPUMLW2E6KRQWJPGQ7ZDAR4EG7DQ, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:19.999711748 (531h51m19.870794783s) [noise=true , address=145.79.227.21:4474 , piece_id=XNT4DNKKZKJMMURNAHTU7HSVQ75S7CELPOG3MXHP4WTHWWB7MO3A, seg_pos={0 0}, upload=466438]: Uploading piece... Dec 27 19:20:21.716426834 (531h51m21.587509879s) [upload=466438]: Next returned nil segment Dec 27 19:20:21.716448456 (531h51m21.587531491s) [upload=466438]: Waiting for error group managing segments... Dec 27 19:20:21.996341062 (531h51m21.867424117s) [noise=true , address=145.79.227.42:4122 , piece_id=YHRGPUUAXUVH6TYL6OOKFCZ7UCL5LBYGQYYMY2J3LUBE3L7RZ6GQ, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.000215580 (531h51m21.871298635s) [noise=true , address=145.79.227.7:4295 , piece_id=MBN4USSWEM7B5TQQFABJWR2U6ZOODV3ABUTACIMJGRG72GZ3WMSQ, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.004316699 (531h51m21.875399734s) [noise=true , address=145.79.227.31:4799 , piece_id=4AENG77AW7IVIEUIMMADTWA3PA45TA6OKAJAWKARWIRV2PT7LQ3Q, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.005037181 (531h51m21.876120226s) [noise=true , address=145.79.227.21:4474 , piece_id=XNT4DNKKZKJMMURNAHTU7HSVQ75S7CELPOG3MXHP4WTHWWB7MO3A, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.009908457 (531h51m21.880991502s) [noise=true , address=145.79.227.17:4972 , piece_id=YA7BKMUB7CD7ZQNWSL5UHSNILEFNE6LGC3XGKMT2XIDNIBT4QF3Q, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.015091867 (531h51m21.886174902s) [noise=true , address=145.79.227.4:4187 , piece_id=KLUHIEALQRM6W36VGJSFRK6YDWU4VC22MIPYFVBAETWR3C6G27JQ, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.019043061 (531h51m21.890126096s) [noise=true , address=145.79.227.50:4374 , piece_id=ZN5RNMMAD7PHGD54DA4CU2RXBKCWFV2ULVO5Q6GEGYLY7FWJ7ESA, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.023649371 (531h51m21.894732407s) [noise=true , address=145.79.227.15:4874 , piece_id=NQTZF44NBHSZNSUKLXW7FVOVXJ7OHDRL577WXHZ2YUVOIOGZC46Q, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.024860217 (531h51m21.895943252s) [noise=true , address=145.79.227.28:4725 , piece_id=SLHZ3AOMVXBF2HIMKCLBB2DRVGVBMWL3PS6TRRAPXD63JMUG6OIQ, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.028326709 (531h51m21.899409744s) [noise=true , address=145.79.227.39:4013 , piece_id=3UAAOUCBBVAJPDXYQP4DVSVZYGA5OMRJ5GKTYP2LNJ4ZBBBJG7HQ, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.174079939 (531h51m22.045162974s) [noise=true , address=77.90.31.21:4878 , piece_id=CTNIAP5SXN7XTHAH5VL2B3XSED7BCHBRQFYYHHNAWN4HDWOEVBSA, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.220873413 (531h51m22.091956448s) [noise=true , address=45.91.116.6:4975 , piece_id=X26CKZUKRFIWYBXVFXRNLWU6C6PBS4NVVNF67OEQX75AVWOKBYHQ, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.233053606 (531h51m22.104136661s) [noise=true , address=45.91.116.12:4227 , piece_id=NXCIPXKWLPDSQYEDSEKHN2FGRNSFNO3CA2F3RTEGP32BRD3R6FMQ, seg_pos={0 0}, upload=466438]: Done uploading piece. err: <nil> Dec 27 19:20:22.233080888 (531h51m22.104163933s) [seg_pos={0 0}, upload=466438]: Segment reached optimal threshold of 13 pieces. Dec 27 19:20:22.233516848 (531h51m22.104599913s) [noise=true , address=77.90.31.10:4518 , piece_id=O3LVLMFDUDB7CKWFHCHM5FQALPUMLW2E6KRQWJPGQ7ZDAR4EG7DQ, seg_pos={0 0}, upload=466438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 19:20:22.233527528 (531h51m22.104610573s) [noise=true , address=77.90.31.10:4518 , piece_id=O3LVLMFDUDB7CKWFHCHM5FQALPUMLW2E6KRQWJPGQ7ZDAR4EG7DQ, seg_pos={0 0}, upload=466438]: Long tail cancellation due to optimal threshold Dec 27 19:20:22.233913042 (531h51m22.104996077s) [noise=true , address=45.91.116.5:4299 , piece_id=GHSCRDWXOMDQLKT4AA7ILJCLPYIHRM6E6Y7CRJDYYBVRKEDXMIPQ, seg_pos={0 0}, upload=466438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 19:20:22.233922901 (531h51m22.105005936s) [noise=true , address=45.91.116.5:4299 , piece_id=GHSCRDWXOMDQLKT4AA7ILJCLPYIHRM6E6Y7CRJDYYBVRKEDXMIPQ, seg_pos={0 0}, upload=466438]: Long tail cancellation due to optimal threshold Dec 27 19:20:22.234150033 (531h51m22.105233078s) [noise=true , address=45.91.116.13:4153 , piece_id=MQXGNMYEARVR5UQXGHLXEEL5EMQFB2RRBZJNJJLYGP4CR6BTOUOQ, seg_pos={0 0}, upload=466438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 27 19:20:22.234160624 (531h51m22.105243669s) [noise=true , address=45.91.116.13:4153 , piece_id=MQXGNMYEARVR5UQXGHLXEEL5EMQFB2RRBZJNJJLYGP4CR6BTOUOQ, seg_pos={0 0}, upload=466438]: Long tail cancellation due to optimal threshold Dec 27 19:20:22.234180832 (531h51m22.105263867s) [seg_pos={0 0}, upload=466438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:20:22.234416862 (531h51m22.105499907s) [upload=466438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:20:22.234425017 (531h51m22.105508052s) [upload=466438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:20:22.525168224 (531h51m22.396251269s) [upload=466438]: Done uploading segments... Dec 27 19:20:22.525192010 (531h51m22.396275055s) [upload=466438]: Upload finished. err: <nil> Duration: 30.002419366s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.46382989s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:19:57 UTC) orbiter warning: context deadline exceeded Duration: 3.463893501s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX.XXXX........XX..XX..X.....X.........X....XXX..XX....X...] duration: 6.241947901s trace: https://jaeger.dev.storj.io/trace/ac7fed860915ad2 FAIL (2025-12-27 19:19:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ) context deadline exceeded ======LOGS====== 27 19:17:28.311502739 (531h48m28.182585784s) [upload=466393]: Starting upload Dec 27 19:17:28.311598922 (531h48m28.182681957s) [upload=466393]: Done starting upload Dec 27 19:17:28.311670709 (531h48m28.182753744s) [upload=466393]: Uploading segments... Dec 27 19:17:28.311853568 (531h48m28.182936613s) [upload=466393]: Got next segment. Inline: false Dec 27 19:17:28.311914834 (531h48m28.182997879s) [upload=466393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:17:28.311921586 (531h48m28.183004621s) [upload=466393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:17:28.556398806 (531h48m28.427481851s) [seg_pos={0 0}, upload=466393]: Begin upload segment... Dec 27 19:17:28.556451646 (531h48m28.427534691s) [seg_pos={0 0}, upload=466393]: Done begin upload segment. Dec 27 19:17:28.556485851 (531h48m28.427568886s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.556513574 (531h48m28.427596609s) [noise=true , address=145.79.227.4:4187 , piece_id=V2GN3FAIWE4PDH26Y4WGUBCGZI74FHFMGEYDUF7TIGAC7VWCGIZA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.556609778 (531h48m28.427692833s) [noise=true , address=77.90.31.10:4518 , piece_id=G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557208237 (531h48m28.428291272s) [noise=true , address=77.90.31.12:4589 , piece_id=4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557447803 (531h48m28.428530869s) [noise=true , address=145.79.227.16:4910 , piece_id=BITFVPWSXB4LG23OWRPM76ZSWFTH2XRI532BN4TUUOHGJSRUFKAA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557794614 (531h48m28.428877649s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557930673 (531h48m28.429013708s) [noise=true , address=145.79.227.6:4257 , piece_id=MQKZYCPGIHOFLIV75VCKNCJMEO5IMJGLG37DZH6I7WCTRYSSLSXQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558313803 (531h48m28.429396838s) [noise=true , address=145.79.227.2:4114 , piece_id=LGLCNKXLCYNUGINZ36KNDG3BYBCZSVD33V3VWSDJESOWQMPJ6Y7Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558354891 (531h48m28.429437936s) [noise=true , address=145.79.227.39:4013 , piece_id=QZFVGTROOUW3ACL74FSNYBIPON6TDDSKGEG3JB7NTQHZH5YNKM3Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558707873 (531h48m28.429790918s) [noise=true , address=145.79.227.28:4727 , piece_id=734F4MHGD6GKD3RKWH2CAVRNCTQKQAP3S7PUPKIMJJVUL2YP6ATA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558725256 (531h48m28.429808301s) [noise=true , address=145.79.227.42:4121 , piece_id=UQIFDM5EUZINBHHNQOU3GMG62AIOPHGQZZAVSEKP2BWA4CAWJBPQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559098246 (531h48m28.430181291s) [noise=true , address=45.91.116.8:4045 , piece_id=RMWS3ESVR4WYBCWJAXSAOZTXWJNKBZYQI2ORZ237IZIPQALXZVDQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559110630 (531h48m28.430193665s) [noise=true , address=145.79.227.36:4518 , piece_id=43XOUW46UQXRGOIWMRR2PCBFTKB5BRUDCAYUDFBHUJPK5F2WBAPQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559496855 (531h48m28.4305799s) [noise=true , address=145.79.227.11:4730 , piece_id=HM7OJGVBTMN3A2LP3LAFARBXF33UZFPKSK7FYQ3GMRLPYWDJCSQQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.561474231 (531h48m28.432557266s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.561987036 (531h48m28.433070082s) [noise=true , address=145.79.227.18:4007 , piece_id=2K4HBUTCJ46T57VFA4ZNPSHB7ISOYMS64VI7ZSJYED5QDVU5XY7Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:30.263658719 (531h48m30.134741764s) [upload=466393]: Next returned nil segment Dec 27 19:17:30.263702041 (531h48m30.134785076s) [upload=466393]: Waiting for error group managing segments... Dec 27 19:17:30.551470414 (531h48m30.422553449s) [noise=true , address=145.79.227.28:4727 , piece_id=734F4MHGD6GKD3RKWH2CAVRNCTQKQAP3S7PUPKIMJJVUL2YP6ATA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.557865431 (531h48m30.428948476s) [noise=true , address=145.79.227.42:4121 , piece_id=UQIFDM5EUZINBHHNQOU3GMG62AIOPHGQZZAVSEKP2BWA4CAWJBPQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.558475773 (531h48m30.429558818s) [noise=true , address=145.79.227.36:4518 , piece_id=43XOUW46UQXRGOIWMRR2PCBFTKB5BRUDCAYUDFBHUJPK5F2WBAPQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.558802806 (531h48m30.429885841s) [noise=true , address=145.79.227.6:4257 , piece_id=MQKZYCPGIHOFLIV75VCKNCJMEO5IMJGLG37DZH6I7WCTRYSSLSXQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.564775680 (531h48m30.435858725s) [noise=true , address=145.79.227.11:4730 , piece_id=HM7OJGVBTMN3A2LP3LAFARBXF33UZFPKSK7FYQ3GMRLPYWDJCSQQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.578674916 (531h48m30.449757951s) [noise=true , address=145.79.227.2:4114 , piece_id=LGLCNKXLCYNUGINZ36KNDG3BYBCZSVD33V3VWSDJESOWQMPJ6Y7Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.579235113 (531h48m30.450318148s) [noise=true , address=145.79.227.39:4013 , piece_id=QZFVGTROOUW3ACL74FSNYBIPON6TDDSKGEG3JB7NTQHZH5YNKM3Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.585965629 (531h48m30.457048664s) [noise=true , address=145.79.227.16:4910 , piece_id=BITFVPWSXB4LG23OWRPM76ZSWFTH2XRI532BN4TUUOHGJSRUFKAA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.603152646 (531h48m30.474235681s) [noise=true , address=145.79.227.18:4007 , piece_id=2K4HBUTCJ46T57VFA4ZNPSHB7ISOYMS64VI7ZSJYED5QDVU5XY7Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.814782097 (531h48m30.685865132s) [noise=true , address=45.91.116.8:4045 , piece_id=RMWS3ESVR4WYBCWJAXSAOZTXWJNKBZYQI2ORZ237IZIPQALXZVDQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.885107349 (531h48m30.756190384s) [noise=true , address=77.90.31.10:4518 , piece_id=G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.936454293 (531h48m30.807537328s) [noise=true , address=145.79.227.4:4187 , piece_id=V2GN3FAIWE4PDH26Y4WGUBCGZI74FHFMGEYDUF7TIGAC7VWCGIZA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:31.664436043 (531h48m31.535519108s) [noise=true , address=77.90.31.12:4589 , piece_id=4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:31.664465107 (531h48m31.535548182s) [seg_pos={0 0}, upload=466393]: Segment reached optimal threshold of 13 pieces. Dec 27 19:17:31.664729311 (531h48m31.535812356s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 27 19:17:31.664738648 (531h48m31.535821683s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.664808691 (531h48m31.535891727s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection Dec 27 19:17:31.664828530 (531h48m31.535911575s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.665017679 (531h48m31.536100714s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 19:17:31.665024563 (531h48m31.536107598s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.665041185 (531h48m31.53612422s) [seg_pos={0 0}, upload=466393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:17:31.665117460 (531h48m31.536200495s) [upload=466393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:17:31.665122469 (531h48m31.536205524s) [upload=466393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:17:31.951558927 (531h48m31.822641972s) [upload=466393]: Done uploading segments... Dec 27 19:17:31.951581399 (531h48m31.822664434s) [upload=466393]: Upload finished. err: <nil> Duration: 6.241973909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XXX.XXXX........XX..XX..X.....X.........X....XXX..XX....X..] duration: 5.709304523s trace: https://jaeger.dev.storj.io/trace/ac7fed860915ad2 FAIL (2025-12-27 19:19:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ) context deadline exceeded ======LOGS====== 27 19:17:28.311502739 (531h48m28.182585784s) [upload=466393]: Starting upload Dec 27 19:17:28.311598922 (531h48m28.182681957s) [upload=466393]: Done starting upload Dec 27 19:17:28.311670709 (531h48m28.182753744s) [upload=466393]: Uploading segments... Dec 27 19:17:28.311853568 (531h48m28.182936613s) [upload=466393]: Got next segment. Inline: false Dec 27 19:17:28.311914834 (531h48m28.182997879s) [upload=466393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:17:28.311921586 (531h48m28.183004621s) [upload=466393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:17:28.556398806 (531h48m28.427481851s) [seg_pos={0 0}, upload=466393]: Begin upload segment... Dec 27 19:17:28.556451646 (531h48m28.427534691s) [seg_pos={0 0}, upload=466393]: Done begin upload segment. Dec 27 19:17:28.556485851 (531h48m28.427568886s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.556513574 (531h48m28.427596609s) [noise=true , address=145.79.227.4:4187 , piece_id=V2GN3FAIWE4PDH26Y4WGUBCGZI74FHFMGEYDUF7TIGAC7VWCGIZA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.556609778 (531h48m28.427692833s) [noise=true , address=77.90.31.10:4518 , piece_id=G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557208237 (531h48m28.428291272s) [noise=true , address=77.90.31.12:4589 , piece_id=4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557447803 (531h48m28.428530869s) [noise=true , address=145.79.227.16:4910 , piece_id=BITFVPWSXB4LG23OWRPM76ZSWFTH2XRI532BN4TUUOHGJSRUFKAA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557794614 (531h48m28.428877649s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557930673 (531h48m28.429013708s) [noise=true , address=145.79.227.6:4257 , piece_id=MQKZYCPGIHOFLIV75VCKNCJMEO5IMJGLG37DZH6I7WCTRYSSLSXQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558313803 (531h48m28.429396838s) [noise=true , address=145.79.227.2:4114 , piece_id=LGLCNKXLCYNUGINZ36KNDG3BYBCZSVD33V3VWSDJESOWQMPJ6Y7Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558354891 (531h48m28.429437936s) [noise=true , address=145.79.227.39:4013 , piece_id=QZFVGTROOUW3ACL74FSNYBIPON6TDDSKGEG3JB7NTQHZH5YNKM3Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558707873 (531h48m28.429790918s) [noise=true , address=145.79.227.28:4727 , piece_id=734F4MHGD6GKD3RKWH2CAVRNCTQKQAP3S7PUPKIMJJVUL2YP6ATA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558725256 (531h48m28.429808301s) [noise=true , address=145.79.227.42:4121 , piece_id=UQIFDM5EUZINBHHNQOU3GMG62AIOPHGQZZAVSEKP2BWA4CAWJBPQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559098246 (531h48m28.430181291s) [noise=true , address=45.91.116.8:4045 , piece_id=RMWS3ESVR4WYBCWJAXSAOZTXWJNKBZYQI2ORZ237IZIPQALXZVDQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559110630 (531h48m28.430193665s) [noise=true , address=145.79.227.36:4518 , piece_id=43XOUW46UQXRGOIWMRR2PCBFTKB5BRUDCAYUDFBHUJPK5F2WBAPQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559496855 (531h48m28.4305799s) [noise=true , address=145.79.227.11:4730 , piece_id=HM7OJGVBTMN3A2LP3LAFARBXF33UZFPKSK7FYQ3GMRLPYWDJCSQQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.561474231 (531h48m28.432557266s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.561987036 (531h48m28.433070082s) [noise=true , address=145.79.227.18:4007 , piece_id=2K4HBUTCJ46T57VFA4ZNPSHB7ISOYMS64VI7ZSJYED5QDVU5XY7Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:30.263658719 (531h48m30.134741764s) [upload=466393]: Next returned nil segment Dec 27 19:17:30.263702041 (531h48m30.134785076s) [upload=466393]: Waiting for error group managing segments... Dec 27 19:17:30.551470414 (531h48m30.422553449s) [noise=true , address=145.79.227.28:4727 , piece_id=734F4MHGD6GKD3RKWH2CAVRNCTQKQAP3S7PUPKIMJJVUL2YP6ATA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.557865431 (531h48m30.428948476s) [noise=true , address=145.79.227.42:4121 , piece_id=UQIFDM5EUZINBHHNQOU3GMG62AIOPHGQZZAVSEKP2BWA4CAWJBPQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.558475773 (531h48m30.429558818s) [noise=true , address=145.79.227.36:4518 , piece_id=43XOUW46UQXRGOIWMRR2PCBFTKB5BRUDCAYUDFBHUJPK5F2WBAPQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.558802806 (531h48m30.429885841s) [noise=true , address=145.79.227.6:4257 , piece_id=MQKZYCPGIHOFLIV75VCKNCJMEO5IMJGLG37DZH6I7WCTRYSSLSXQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.564775680 (531h48m30.435858725s) [noise=true , address=145.79.227.11:4730 , piece_id=HM7OJGVBTMN3A2LP3LAFARBXF33UZFPKSK7FYQ3GMRLPYWDJCSQQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.578674916 (531h48m30.449757951s) [noise=true , address=145.79.227.2:4114 , piece_id=LGLCNKXLCYNUGINZ36KNDG3BYBCZSVD33V3VWSDJESOWQMPJ6Y7Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.579235113 (531h48m30.450318148s) [noise=true , address=145.79.227.39:4013 , piece_id=QZFVGTROOUW3ACL74FSNYBIPON6TDDSKGEG3JB7NTQHZH5YNKM3Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.585965629 (531h48m30.457048664s) [noise=true , address=145.79.227.16:4910 , piece_id=BITFVPWSXB4LG23OWRPM76ZSWFTH2XRI532BN4TUUOHGJSRUFKAA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.603152646 (531h48m30.474235681s) [noise=true , address=145.79.227.18:4007 , piece_id=2K4HBUTCJ46T57VFA4ZNPSHB7ISOYMS64VI7ZSJYED5QDVU5XY7Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.814782097 (531h48m30.685865132s) [noise=true , address=45.91.116.8:4045 , piece_id=RMWS3ESVR4WYBCWJAXSAOZTXWJNKBZYQI2ORZ237IZIPQALXZVDQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.885107349 (531h48m30.756190384s) [noise=true , address=77.90.31.10:4518 , piece_id=G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.936454293 (531h48m30.807537328s) [noise=true , address=145.79.227.4:4187 , piece_id=V2GN3FAIWE4PDH26Y4WGUBCGZI74FHFMGEYDUF7TIGAC7VWCGIZA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:31.664436043 (531h48m31.535519108s) [noise=true , address=77.90.31.12:4589 , piece_id=4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:31.664465107 (531h48m31.535548182s) [seg_pos={0 0}, upload=466393]: Segment reached optimal threshold of 13 pieces. Dec 27 19:17:31.664729311 (531h48m31.535812356s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 27 19:17:31.664738648 (531h48m31.535821683s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.664808691 (531h48m31.535891727s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection Dec 27 19:17:31.664828530 (531h48m31.535911575s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.665017679 (531h48m31.536100714s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 19:17:31.665024563 (531h48m31.536107598s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.665041185 (531h48m31.53612422s) [seg_pos={0 0}, upload=466393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:17:31.665117460 (531h48m31.536200495s) [upload=466393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:17:31.665122469 (531h48m31.536205524s) [upload=466393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:17:31.951558927 (531h48m31.822641972s) [upload=466393]: Done uploading segments... Dec 27 19:17:31.951581399 (531h48m31.822664434s) [upload=466393]: Upload finished. err: <nil> Duration: 5.709333608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.503482848s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:18:49 UTC) orbiter warning: context deadline exceeded Duration: 3.503511784s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XXX.XXXX........XX..XX..X.....X.........X....XXX..XX....X.] duration: 26.455156064s trace: https://jaeger.dev.storj.io/trace/ac7fed860915ad2 FAIL (2025-12-27 19:18:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ) context deadline exceeded ======LOGS====== 27 19:17:28.311502739 (531h48m28.182585784s) [upload=466393]: Starting upload Dec 27 19:17:28.311598922 (531h48m28.182681957s) [upload=466393]: Done starting upload Dec 27 19:17:28.311670709 (531h48m28.182753744s) [upload=466393]: Uploading segments... Dec 27 19:17:28.311853568 (531h48m28.182936613s) [upload=466393]: Got next segment. Inline: false Dec 27 19:17:28.311914834 (531h48m28.182997879s) [upload=466393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:17:28.311921586 (531h48m28.183004621s) [upload=466393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:17:28.556398806 (531h48m28.427481851s) [seg_pos={0 0}, upload=466393]: Begin upload segment... Dec 27 19:17:28.556451646 (531h48m28.427534691s) [seg_pos={0 0}, upload=466393]: Done begin upload segment. Dec 27 19:17:28.556485851 (531h48m28.427568886s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.556513574 (531h48m28.427596609s) [noise=true , address=145.79.227.4:4187 , piece_id=V2GN3FAIWE4PDH26Y4WGUBCGZI74FHFMGEYDUF7TIGAC7VWCGIZA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.556609778 (531h48m28.427692833s) [noise=true , address=77.90.31.10:4518 , piece_id=G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557208237 (531h48m28.428291272s) [noise=true , address=77.90.31.12:4589 , piece_id=4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557447803 (531h48m28.428530869s) [noise=true , address=145.79.227.16:4910 , piece_id=BITFVPWSXB4LG23OWRPM76ZSWFTH2XRI532BN4TUUOHGJSRUFKAA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557794614 (531h48m28.428877649s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557930673 (531h48m28.429013708s) [noise=true , address=145.79.227.6:4257 , piece_id=MQKZYCPGIHOFLIV75VCKNCJMEO5IMJGLG37DZH6I7WCTRYSSLSXQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558313803 (531h48m28.429396838s) [noise=true , address=145.79.227.2:4114 , piece_id=LGLCNKXLCYNUGINZ36KNDG3BYBCZSVD33V3VWSDJESOWQMPJ6Y7Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558354891 (531h48m28.429437936s) [noise=true , address=145.79.227.39:4013 , piece_id=QZFVGTROOUW3ACL74FSNYBIPON6TDDSKGEG3JB7NTQHZH5YNKM3Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558707873 (531h48m28.429790918s) [noise=true , address=145.79.227.28:4727 , piece_id=734F4MHGD6GKD3RKWH2CAVRNCTQKQAP3S7PUPKIMJJVUL2YP6ATA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558725256 (531h48m28.429808301s) [noise=true , address=145.79.227.42:4121 , piece_id=UQIFDM5EUZINBHHNQOU3GMG62AIOPHGQZZAVSEKP2BWA4CAWJBPQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559098246 (531h48m28.430181291s) [noise=true , address=45.91.116.8:4045 , piece_id=RMWS3ESVR4WYBCWJAXSAOZTXWJNKBZYQI2ORZ237IZIPQALXZVDQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559110630 (531h48m28.430193665s) [noise=true , address=145.79.227.36:4518 , piece_id=43XOUW46UQXRGOIWMRR2PCBFTKB5BRUDCAYUDFBHUJPK5F2WBAPQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559496855 (531h48m28.4305799s) [noise=true , address=145.79.227.11:4730 , piece_id=HM7OJGVBTMN3A2LP3LAFARBXF33UZFPKSK7FYQ3GMRLPYWDJCSQQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.561474231 (531h48m28.432557266s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.561987036 (531h48m28.433070082s) [noise=true , address=145.79.227.18:4007 , piece_id=2K4HBUTCJ46T57VFA4ZNPSHB7ISOYMS64VI7ZSJYED5QDVU5XY7Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:30.263658719 (531h48m30.134741764s) [upload=466393]: Next returned nil segment Dec 27 19:17:30.263702041 (531h48m30.134785076s) [upload=466393]: Waiting for error group managing segments... Dec 27 19:17:30.551470414 (531h48m30.422553449s) [noise=true , address=145.79.227.28:4727 , piece_id=734F4MHGD6GKD3RKWH2CAVRNCTQKQAP3S7PUPKIMJJVUL2YP6ATA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.557865431 (531h48m30.428948476s) [noise=true , address=145.79.227.42:4121 , piece_id=UQIFDM5EUZINBHHNQOU3GMG62AIOPHGQZZAVSEKP2BWA4CAWJBPQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.558475773 (531h48m30.429558818s) [noise=true , address=145.79.227.36:4518 , piece_id=43XOUW46UQXRGOIWMRR2PCBFTKB5BRUDCAYUDFBHUJPK5F2WBAPQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.558802806 (531h48m30.429885841s) [noise=true , address=145.79.227.6:4257 , piece_id=MQKZYCPGIHOFLIV75VCKNCJMEO5IMJGLG37DZH6I7WCTRYSSLSXQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.564775680 (531h48m30.435858725s) [noise=true , address=145.79.227.11:4730 , piece_id=HM7OJGVBTMN3A2LP3LAFARBXF33UZFPKSK7FYQ3GMRLPYWDJCSQQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.578674916 (531h48m30.449757951s) [noise=true , address=145.79.227.2:4114 , piece_id=LGLCNKXLCYNUGINZ36KNDG3BYBCZSVD33V3VWSDJESOWQMPJ6Y7Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.579235113 (531h48m30.450318148s) [noise=true , address=145.79.227.39:4013 , piece_id=QZFVGTROOUW3ACL74FSNYBIPON6TDDSKGEG3JB7NTQHZH5YNKM3Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.585965629 (531h48m30.457048664s) [noise=true , address=145.79.227.16:4910 , piece_id=BITFVPWSXB4LG23OWRPM76ZSWFTH2XRI532BN4TUUOHGJSRUFKAA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.603152646 (531h48m30.474235681s) [noise=true , address=145.79.227.18:4007 , piece_id=2K4HBUTCJ46T57VFA4ZNPSHB7ISOYMS64VI7ZSJYED5QDVU5XY7Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.814782097 (531h48m30.685865132s) [noise=true , address=45.91.116.8:4045 , piece_id=RMWS3ESVR4WYBCWJAXSAOZTXWJNKBZYQI2ORZ237IZIPQALXZVDQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.885107349 (531h48m30.756190384s) [noise=true , address=77.90.31.10:4518 , piece_id=G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.936454293 (531h48m30.807537328s) [noise=true , address=145.79.227.4:4187 , piece_id=V2GN3FAIWE4PDH26Y4WGUBCGZI74FHFMGEYDUF7TIGAC7VWCGIZA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:31.664436043 (531h48m31.535519108s) [noise=true , address=77.90.31.12:4589 , piece_id=4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:31.664465107 (531h48m31.535548182s) [seg_pos={0 0}, upload=466393]: Segment reached optimal threshold of 13 pieces. Dec 27 19:17:31.664729311 (531h48m31.535812356s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 27 19:17:31.664738648 (531h48m31.535821683s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.664808691 (531h48m31.535891727s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection Dec 27 19:17:31.664828530 (531h48m31.535911575s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.665017679 (531h48m31.536100714s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 19:17:31.665024563 (531h48m31.536107598s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.665041185 (531h48m31.53612422s) [seg_pos={0 0}, upload=466393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:17:31.665117460 (531h48m31.536200495s) [upload=466393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:17:31.665122469 (531h48m31.536205524s) [upload=466393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:17:31.951558927 (531h48m31.822641972s) [upload=466393]: Done uploading segments... Dec 27 19:17:31.951581399 (531h48m31.822664434s) [upload=466393]: Upload finished. err: <nil> Duration: 26.455179519s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.560996475s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:18:33 UTC) orbiter warning: context deadline exceeded Duration: 3.561022897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..XXX.XXXX........XX..XX..X.....X.........X....XXX..XX....X] duration: 30.002146055s trace: https://jaeger.dev.storj.io/trace/ac7fed860915ad2 FAIL (2025-12-27 19:17:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ) context deadline exceeded ======LOGS====== 27 19:17:28.311502739 (531h48m28.182585784s) [upload=466393]: Starting upload Dec 27 19:17:28.311598922 (531h48m28.182681957s) [upload=466393]: Done starting upload Dec 27 19:17:28.311670709 (531h48m28.182753744s) [upload=466393]: Uploading segments... Dec 27 19:17:28.311853568 (531h48m28.182936613s) [upload=466393]: Got next segment. Inline: false Dec 27 19:17:28.311914834 (531h48m28.182997879s) [upload=466393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:17:28.311921586 (531h48m28.183004621s) [upload=466393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:17:28.556398806 (531h48m28.427481851s) [seg_pos={0 0}, upload=466393]: Begin upload segment... Dec 27 19:17:28.556451646 (531h48m28.427534691s) [seg_pos={0 0}, upload=466393]: Done begin upload segment. Dec 27 19:17:28.556485851 (531h48m28.427568886s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.556513574 (531h48m28.427596609s) [noise=true , address=145.79.227.4:4187 , piece_id=V2GN3FAIWE4PDH26Y4WGUBCGZI74FHFMGEYDUF7TIGAC7VWCGIZA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.556609778 (531h48m28.427692833s) [noise=true , address=77.90.31.10:4518 , piece_id=G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557208237 (531h48m28.428291272s) [noise=true , address=77.90.31.12:4589 , piece_id=4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557447803 (531h48m28.428530869s) [noise=true , address=145.79.227.16:4910 , piece_id=BITFVPWSXB4LG23OWRPM76ZSWFTH2XRI532BN4TUUOHGJSRUFKAA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557794614 (531h48m28.428877649s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.557930673 (531h48m28.429013708s) [noise=true , address=145.79.227.6:4257 , piece_id=MQKZYCPGIHOFLIV75VCKNCJMEO5IMJGLG37DZH6I7WCTRYSSLSXQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558313803 (531h48m28.429396838s) [noise=true , address=145.79.227.2:4114 , piece_id=LGLCNKXLCYNUGINZ36KNDG3BYBCZSVD33V3VWSDJESOWQMPJ6Y7Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558354891 (531h48m28.429437936s) [noise=true , address=145.79.227.39:4013 , piece_id=QZFVGTROOUW3ACL74FSNYBIPON6TDDSKGEG3JB7NTQHZH5YNKM3Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558707873 (531h48m28.429790918s) [noise=true , address=145.79.227.28:4727 , piece_id=734F4MHGD6GKD3RKWH2CAVRNCTQKQAP3S7PUPKIMJJVUL2YP6ATA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.558725256 (531h48m28.429808301s) [noise=true , address=145.79.227.42:4121 , piece_id=UQIFDM5EUZINBHHNQOU3GMG62AIOPHGQZZAVSEKP2BWA4CAWJBPQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559098246 (531h48m28.430181291s) [noise=true , address=45.91.116.8:4045 , piece_id=RMWS3ESVR4WYBCWJAXSAOZTXWJNKBZYQI2ORZ237IZIPQALXZVDQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559110630 (531h48m28.430193665s) [noise=true , address=145.79.227.36:4518 , piece_id=43XOUW46UQXRGOIWMRR2PCBFTKB5BRUDCAYUDFBHUJPK5F2WBAPQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.559496855 (531h48m28.4305799s) [noise=true , address=145.79.227.11:4730 , piece_id=HM7OJGVBTMN3A2LP3LAFARBXF33UZFPKSK7FYQ3GMRLPYWDJCSQQ, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.561474231 (531h48m28.432557266s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:28.561987036 (531h48m28.433070082s) [noise=true , address=145.79.227.18:4007 , piece_id=2K4HBUTCJ46T57VFA4ZNPSHB7ISOYMS64VI7ZSJYED5QDVU5XY7Q, seg_pos={0 0}, upload=466393]: Uploading piece... Dec 27 19:17:30.263658719 (531h48m30.134741764s) [upload=466393]: Next returned nil segment Dec 27 19:17:30.263702041 (531h48m30.134785076s) [upload=466393]: Waiting for error group managing segments... Dec 27 19:17:30.551470414 (531h48m30.422553449s) [noise=true , address=145.79.227.28:4727 , piece_id=734F4MHGD6GKD3RKWH2CAVRNCTQKQAP3S7PUPKIMJJVUL2YP6ATA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.557865431 (531h48m30.428948476s) [noise=true , address=145.79.227.42:4121 , piece_id=UQIFDM5EUZINBHHNQOU3GMG62AIOPHGQZZAVSEKP2BWA4CAWJBPQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.558475773 (531h48m30.429558818s) [noise=true , address=145.79.227.36:4518 , piece_id=43XOUW46UQXRGOIWMRR2PCBFTKB5BRUDCAYUDFBHUJPK5F2WBAPQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.558802806 (531h48m30.429885841s) [noise=true , address=145.79.227.6:4257 , piece_id=MQKZYCPGIHOFLIV75VCKNCJMEO5IMJGLG37DZH6I7WCTRYSSLSXQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.564775680 (531h48m30.435858725s) [noise=true , address=145.79.227.11:4730 , piece_id=HM7OJGVBTMN3A2LP3LAFARBXF33UZFPKSK7FYQ3GMRLPYWDJCSQQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.578674916 (531h48m30.449757951s) [noise=true , address=145.79.227.2:4114 , piece_id=LGLCNKXLCYNUGINZ36KNDG3BYBCZSVD33V3VWSDJESOWQMPJ6Y7Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.579235113 (531h48m30.450318148s) [noise=true , address=145.79.227.39:4013 , piece_id=QZFVGTROOUW3ACL74FSNYBIPON6TDDSKGEG3JB7NTQHZH5YNKM3Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.585965629 (531h48m30.457048664s) [noise=true , address=145.79.227.16:4910 , piece_id=BITFVPWSXB4LG23OWRPM76ZSWFTH2XRI532BN4TUUOHGJSRUFKAA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.603152646 (531h48m30.474235681s) [noise=true , address=145.79.227.18:4007 , piece_id=2K4HBUTCJ46T57VFA4ZNPSHB7ISOYMS64VI7ZSJYED5QDVU5XY7Q, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.814782097 (531h48m30.685865132s) [noise=true , address=45.91.116.8:4045 , piece_id=RMWS3ESVR4WYBCWJAXSAOZTXWJNKBZYQI2ORZ237IZIPQALXZVDQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.885107349 (531h48m30.756190384s) [noise=true , address=77.90.31.10:4518 , piece_id=G3LD5Z7UXNCTRJFAJDPP5MT4D76ZMCVA3UJV37HQT3MGKXN3NSSA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:30.936454293 (531h48m30.807537328s) [noise=true , address=145.79.227.4:4187 , piece_id=V2GN3FAIWE4PDH26Y4WGUBCGZI74FHFMGEYDUF7TIGAC7VWCGIZA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:31.664436043 (531h48m31.535519108s) [noise=true , address=77.90.31.12:4589 , piece_id=4IQNYHNJGPE2N6NJRZEXU65TXH3FAKYL4WSOXZTVUATQWSTHTLGQ, seg_pos={0 0}, upload=466393]: Done uploading piece. err: <nil> Dec 27 19:17:31.664465107 (531h48m31.535548182s) [seg_pos={0 0}, upload=466393]: Segment reached optimal threshold of 13 pieces. Dec 27 19:17:31.664729311 (531h48m31.535812356s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF Dec 27 19:17:31.664738648 (531h48m31.535821683s) [noise=true , address=145.79.227.49:4337 , piece_id=3VQYAINFLCW5HNMFHTZCAX6TEREENGKLLBBFYZWIZAQGGBEBPEHA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.664808691 (531h48m31.535891727s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38844->77.90.31.18:4808: use of closed network connection Dec 27 19:17:31.664828530 (531h48m31.535911575s) [noise=true , address=77.90.31.18:4808 , piece_id=VQYH7LXTG2YMLWFNRIBZGVUHJYEMSJOXMPRU37D3R7SJQGHIF7YA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.665017679 (531h48m31.536100714s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 19:17:31.665024563 (531h48m31.536107598s) [noise=true , address=45.91.116.9:4263 , piece_id=PJKJAVBBDAFLUL2777UEQ27XS743W2HZ7ZKAMGBF2TCIHRGSGBHA, seg_pos={0 0}, upload=466393]: Long tail cancellation due to optimal threshold Dec 27 19:17:31.665041185 (531h48m31.53612422s) [seg_pos={0 0}, upload=466393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:17:31.665117460 (531h48m31.536200495s) [upload=466393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:17:31.665122469 (531h48m31.536205524s) [upload=466393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:17:31.951558927 (531h48m31.822641972s) [upload=466393]: Done uploading segments... Dec 27 19:17:31.951581399 (531h48m31.822664434s) [upload=466393]: Upload finished. err: <nil> Duration: 30.002157808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX..XXX.XXXX........XX..XX..X.....X.........X....XXX..XX....] duration: 28.494376609s trace: https://jaeger.dev.storj.io/trace/3156bafcda88c4d5 FAIL (2025-12-27 19:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ) context deadline exceeded ======LOGS====== ng segments... Dec 27 19:13:26.098311945 (531h44m25.96939499s) [upload=466337]: Got next segment. Inline: false Dec 27 19:13:26.098363603 (531h44m25.969446638s) [upload=466337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:13:26.098369896 (531h44m25.969452931s) [upload=466337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:13:26.376014842 (531h44m26.247097877s) [seg_pos={0 0}, upload=466337]: Begin upload segment... Dec 27 19:13:26.376079475 (531h44m26.24716252s) [seg_pos={0 0}, upload=466337]: Done begin upload segment. Dec 27 19:13:26.376088552 (531h44m26.247171597s) [noise=true , address=145.79.227.5:4223 , piece_id=T4TVGCCPE5BPIJ3K35QDTNNGMTQTJRHI7KSCNYC6B2DZSDAKDIHA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.376114982 (531h44m26.247198038s) [noise=true , address=45.91.116.11:4117 , piece_id=NDAPJYPSODZPJHF4HIC5NAR4WV5USXKG3VHTKNRFNZPXUZ4SIDMA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.376932278 (531h44m26.248015313s) [noise=true , address=145.79.227.47:4265 , piece_id=WTSCDTBRSDO2Y3MQL4PZ4KP2PO4J3CEP2Z2LCWHYI7J7Y3D7LITQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.377325036 (531h44m26.248408071s) [noise=true , address=145.79.227.50:4374 , piece_id=3B6JSKR2U2RYNZLJPYW54Q4HXNGDCPBHUOQX5VNCWLDGCE35B4FQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.377657509 (531h44m26.248740544s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378008057 (531h44m26.249091092s) [noise=true , address=77.90.31.14:4661 , piece_id=OQTIZVTVXGONOJQ2XU2QKBGMKXQ6ZUATU7ANYOVKWLJIPHVZHBHQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378490385 (531h44m26.24957343s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378802850 (531h44m26.249885885s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.379142387 (531h44m26.250225422s) [noise=true , address=145.79.227.2:4113 , piece_id=MGBWZ7FDHLWM2TD3CDQ7VHUMLXUEOHWLR5P64FRAVE5QUK52VYDQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.379476383 (531h44m26.250559418s) [noise=true , address=77.90.31.10:4519 , piece_id=7YTYI2KH6D6Y2PGHROPP7OZATMF6QSL5QZCGIYXJOHWJBWM3DOYA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.380608759 (531h44m26.251691804s) [noise=true , address=145.79.227.48:4302 , piece_id=KU4RA7PQR3F5KJA55SZ2VIUDHPFOGX7XRECZJECIOEAWFLICP5MA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.380676458 (531h44m26.251759493s) [noise=true , address=45.91.116.13:4155 , piece_id=MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.381659790 (531h44m26.252742836s) [noise=true , address=145.79.227.9:4365 , piece_id=3FWWZUSNRBMFGHVALMQRAIFAANZT2QLZOIJVSGWBTOG3VL5CYGNQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.382353291 (531h44m26.253436336s) [noise=true , address=145.79.227.11:4732 , piece_id=AVL3Q7TMRHWGNZ22TYG5IPXIRMVEFJY4W3ULIUPUPI23NLFSSXGA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.383714052 (531h44m26.254797097s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.385082849 (531h44m26.256165894s) [noise=true , address=145.79.227.37:4565 , piece_id=HFM74QOLJTELRMBPFG4PWLXMQOBZD5SLSMQZCGOSTZXBBBKJHKGA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:28.089194807 (531h44m27.960277842s) [upload=466337]: Next returned nil segment Dec 27 19:13:28.089217240 (531h44m27.960300275s) [upload=466337]: Waiting for error group managing segments... Dec 27 19:13:28.387646493 (531h44m28.258729528s) [noise=true , address=145.79.227.48:4302 , piece_id=KU4RA7PQR3F5KJA55SZ2VIUDHPFOGX7XRECZJECIOEAWFLICP5MA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.388527130 (531h44m28.259610166s) [noise=true , address=145.79.227.37:4565 , piece_id=HFM74QOLJTELRMBPFG4PWLXMQOBZD5SLSMQZCGOSTZXBBBKJHKGA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.389251960 (531h44m28.260334995s) [noise=true , address=145.79.227.5:4223 , piece_id=T4TVGCCPE5BPIJ3K35QDTNNGMTQTJRHI7KSCNYC6B2DZSDAKDIHA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.395362295 (531h44m28.266445381s) [noise=true , address=145.79.227.2:4113 , piece_id=MGBWZ7FDHLWM2TD3CDQ7VHUMLXUEOHWLR5P64FRAVE5QUK52VYDQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.398843915 (531h44m28.26992696s) [noise=true , address=145.79.227.50:4374 , piece_id=3B6JSKR2U2RYNZLJPYW54Q4HXNGDCPBHUOQX5VNCWLDGCE35B4FQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.406970360 (531h44m28.278053395s) [noise=true , address=145.79.227.47:4265 , piece_id=WTSCDTBRSDO2Y3MQL4PZ4KP2PO4J3CEP2Z2LCWHYI7J7Y3D7LITQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.407352808 (531h44m28.278435853s) [noise=true , address=145.79.227.11:4732 , piece_id=AVL3Q7TMRHWGNZ22TYG5IPXIRMVEFJY4W3ULIUPUPI23NLFSSXGA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.416185658 (531h44m28.287268703s) [noise=true , address=145.79.227.9:4365 , piece_id=3FWWZUSNRBMFGHVALMQRAIFAANZT2QLZOIJVSGWBTOG3VL5CYGNQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.592079671 (531h44m28.463162706s) [noise=true , address=45.91.116.11:4117 , piece_id=NDAPJYPSODZPJHF4HIC5NAR4WV5USXKG3VHTKNRFNZPXUZ4SIDMA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.637960927 (531h44m28.509043972s) [noise=true , address=77.90.31.10:4519 , piece_id=7YTYI2KH6D6Y2PGHROPP7OZATMF6QSL5QZCGIYXJOHWJBWM3DOYA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.657276147 (531h44m28.528359182s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.681990348 (531h44m28.553073383s) [noise=true , address=77.90.31.14:4661 , piece_id=OQTIZVTVXGONOJQ2XU2QKBGMKXQ6ZUATU7ANYOVKWLJIPHVZHBHQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.734861374 (531h44m28.605944429s) [noise=true , address=45.91.116.13:4155 , piece_id=MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.734881822 (531h44m28.605964867s) [seg_pos={0 0}, upload=466337]: Segment reached optimal threshold of 13 pieces. Dec 27 19:13:28.735090229 (531h44m28.606173264s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection Dec 27 19:13:28.735097934 (531h44m28.606180969s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735215498 (531h44m28.606298533s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection Dec 27 19:13:28.735236849 (531h44m28.606319884s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735340145 (531h44m28.606423181s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 19:13:28.735347209 (531h44m28.606430244s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735365884 (531h44m28.606448919s) [seg_pos={0 0}, upload=466337]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:13:28.735450015 (531h44m28.60653305s) [upload=466337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:13:28.735456547 (531h44m28.606539582s) [upload=466337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:13:29.034869243 (531h44m28.905952278s) [upload=466337]: Done uploading segments... Dec 27 19:13:29.034889040 (531h44m28.905972085s) [upload=466337]: Upload finished. err: <nil> Duration: 28.494423308s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.551869524s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:17:20 UTC) orbiter warning: context deadline exceeded Duration: 3.551898008s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.494610944s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:16:44 UTC) orbiter warning: context deadline exceeded Duration: 3.494730613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX..XXX.XXXX........XX..XX..X.....X.........X....XXX..XX...] duration: 6.032056017s trace: https://jaeger.dev.storj.io/trace/3156bafcda88c4d5 FAIL (2025-12-27 19:16:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ) context deadline exceeded ======LOGS====== ng segments... Dec 27 19:13:26.098311945 (531h44m25.96939499s) [upload=466337]: Got next segment. Inline: false Dec 27 19:13:26.098363603 (531h44m25.969446638s) [upload=466337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:13:26.098369896 (531h44m25.969452931s) [upload=466337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:13:26.376014842 (531h44m26.247097877s) [seg_pos={0 0}, upload=466337]: Begin upload segment... Dec 27 19:13:26.376079475 (531h44m26.24716252s) [seg_pos={0 0}, upload=466337]: Done begin upload segment. Dec 27 19:13:26.376088552 (531h44m26.247171597s) [noise=true , address=145.79.227.5:4223 , piece_id=T4TVGCCPE5BPIJ3K35QDTNNGMTQTJRHI7KSCNYC6B2DZSDAKDIHA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.376114982 (531h44m26.247198038s) [noise=true , address=45.91.116.11:4117 , piece_id=NDAPJYPSODZPJHF4HIC5NAR4WV5USXKG3VHTKNRFNZPXUZ4SIDMA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.376932278 (531h44m26.248015313s) [noise=true , address=145.79.227.47:4265 , piece_id=WTSCDTBRSDO2Y3MQL4PZ4KP2PO4J3CEP2Z2LCWHYI7J7Y3D7LITQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.377325036 (531h44m26.248408071s) [noise=true , address=145.79.227.50:4374 , piece_id=3B6JSKR2U2RYNZLJPYW54Q4HXNGDCPBHUOQX5VNCWLDGCE35B4FQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.377657509 (531h44m26.248740544s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378008057 (531h44m26.249091092s) [noise=true , address=77.90.31.14:4661 , piece_id=OQTIZVTVXGONOJQ2XU2QKBGMKXQ6ZUATU7ANYOVKWLJIPHVZHBHQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378490385 (531h44m26.24957343s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378802850 (531h44m26.249885885s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.379142387 (531h44m26.250225422s) [noise=true , address=145.79.227.2:4113 , piece_id=MGBWZ7FDHLWM2TD3CDQ7VHUMLXUEOHWLR5P64FRAVE5QUK52VYDQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.379476383 (531h44m26.250559418s) [noise=true , address=77.90.31.10:4519 , piece_id=7YTYI2KH6D6Y2PGHROPP7OZATMF6QSL5QZCGIYXJOHWJBWM3DOYA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.380608759 (531h44m26.251691804s) [noise=true , address=145.79.227.48:4302 , piece_id=KU4RA7PQR3F5KJA55SZ2VIUDHPFOGX7XRECZJECIOEAWFLICP5MA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.380676458 (531h44m26.251759493s) [noise=true , address=45.91.116.13:4155 , piece_id=MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.381659790 (531h44m26.252742836s) [noise=true , address=145.79.227.9:4365 , piece_id=3FWWZUSNRBMFGHVALMQRAIFAANZT2QLZOIJVSGWBTOG3VL5CYGNQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.382353291 (531h44m26.253436336s) [noise=true , address=145.79.227.11:4732 , piece_id=AVL3Q7TMRHWGNZ22TYG5IPXIRMVEFJY4W3ULIUPUPI23NLFSSXGA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.383714052 (531h44m26.254797097s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.385082849 (531h44m26.256165894s) [noise=true , address=145.79.227.37:4565 , piece_id=HFM74QOLJTELRMBPFG4PWLXMQOBZD5SLSMQZCGOSTZXBBBKJHKGA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:28.089194807 (531h44m27.960277842s) [upload=466337]: Next returned nil segment Dec 27 19:13:28.089217240 (531h44m27.960300275s) [upload=466337]: Waiting for error group managing segments... Dec 27 19:13:28.387646493 (531h44m28.258729528s) [noise=true , address=145.79.227.48:4302 , piece_id=KU4RA7PQR3F5KJA55SZ2VIUDHPFOGX7XRECZJECIOEAWFLICP5MA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.388527130 (531h44m28.259610166s) [noise=true , address=145.79.227.37:4565 , piece_id=HFM74QOLJTELRMBPFG4PWLXMQOBZD5SLSMQZCGOSTZXBBBKJHKGA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.389251960 (531h44m28.260334995s) [noise=true , address=145.79.227.5:4223 , piece_id=T4TVGCCPE5BPIJ3K35QDTNNGMTQTJRHI7KSCNYC6B2DZSDAKDIHA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.395362295 (531h44m28.266445381s) [noise=true , address=145.79.227.2:4113 , piece_id=MGBWZ7FDHLWM2TD3CDQ7VHUMLXUEOHWLR5P64FRAVE5QUK52VYDQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.398843915 (531h44m28.26992696s) [noise=true , address=145.79.227.50:4374 , piece_id=3B6JSKR2U2RYNZLJPYW54Q4HXNGDCPBHUOQX5VNCWLDGCE35B4FQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.406970360 (531h44m28.278053395s) [noise=true , address=145.79.227.47:4265 , piece_id=WTSCDTBRSDO2Y3MQL4PZ4KP2PO4J3CEP2Z2LCWHYI7J7Y3D7LITQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.407352808 (531h44m28.278435853s) [noise=true , address=145.79.227.11:4732 , piece_id=AVL3Q7TMRHWGNZ22TYG5IPXIRMVEFJY4W3ULIUPUPI23NLFSSXGA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.416185658 (531h44m28.287268703s) [noise=true , address=145.79.227.9:4365 , piece_id=3FWWZUSNRBMFGHVALMQRAIFAANZT2QLZOIJVSGWBTOG3VL5CYGNQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.592079671 (531h44m28.463162706s) [noise=true , address=45.91.116.11:4117 , piece_id=NDAPJYPSODZPJHF4HIC5NAR4WV5USXKG3VHTKNRFNZPXUZ4SIDMA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.637960927 (531h44m28.509043972s) [noise=true , address=77.90.31.10:4519 , piece_id=7YTYI2KH6D6Y2PGHROPP7OZATMF6QSL5QZCGIYXJOHWJBWM3DOYA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.657276147 (531h44m28.528359182s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.681990348 (531h44m28.553073383s) [noise=true , address=77.90.31.14:4661 , piece_id=OQTIZVTVXGONOJQ2XU2QKBGMKXQ6ZUATU7ANYOVKWLJIPHVZHBHQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.734861374 (531h44m28.605944429s) [noise=true , address=45.91.116.13:4155 , piece_id=MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.734881822 (531h44m28.605964867s) [seg_pos={0 0}, upload=466337]: Segment reached optimal threshold of 13 pieces. Dec 27 19:13:28.735090229 (531h44m28.606173264s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection Dec 27 19:13:28.735097934 (531h44m28.606180969s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735215498 (531h44m28.606298533s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection Dec 27 19:13:28.735236849 (531h44m28.606319884s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735340145 (531h44m28.606423181s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 19:13:28.735347209 (531h44m28.606430244s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735365884 (531h44m28.606448919s) [seg_pos={0 0}, upload=466337]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:13:28.735450015 (531h44m28.60653305s) [upload=466337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:13:28.735456547 (531h44m28.606539582s) [upload=466337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:13:29.034869243 (531h44m28.905952278s) [upload=466337]: Done uploading segments... Dec 27 19:13:29.034889040 (531h44m28.905972085s) [upload=466337]: Upload finished. err: <nil> Duration: 6.032089842s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001710143s trace: https://jaeger.dev.storj.io/trace/4e0ce2c07c03024e FAIL (2025-12-27 19:15:37 UTC) orbiter warning: context deadline exceeded Duration: 5.001806506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXX..XXX.XXXX........XX..XX..X.....X.........X....XXX..XX..] duration: 13.839062536s trace: https://jaeger.dev.storj.io/trace/3156bafcda88c4d5 FAIL (2025-12-27 19:15:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ) context deadline exceeded ======LOGS====== ng segments... Dec 27 19:13:26.098311945 (531h44m25.96939499s) [upload=466337]: Got next segment. Inline: false Dec 27 19:13:26.098363603 (531h44m25.969446638s) [upload=466337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:13:26.098369896 (531h44m25.969452931s) [upload=466337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:13:26.376014842 (531h44m26.247097877s) [seg_pos={0 0}, upload=466337]: Begin upload segment... Dec 27 19:13:26.376079475 (531h44m26.24716252s) [seg_pos={0 0}, upload=466337]: Done begin upload segment. Dec 27 19:13:26.376088552 (531h44m26.247171597s) [noise=true , address=145.79.227.5:4223 , piece_id=T4TVGCCPE5BPIJ3K35QDTNNGMTQTJRHI7KSCNYC6B2DZSDAKDIHA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.376114982 (531h44m26.247198038s) [noise=true , address=45.91.116.11:4117 , piece_id=NDAPJYPSODZPJHF4HIC5NAR4WV5USXKG3VHTKNRFNZPXUZ4SIDMA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.376932278 (531h44m26.248015313s) [noise=true , address=145.79.227.47:4265 , piece_id=WTSCDTBRSDO2Y3MQL4PZ4KP2PO4J3CEP2Z2LCWHYI7J7Y3D7LITQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.377325036 (531h44m26.248408071s) [noise=true , address=145.79.227.50:4374 , piece_id=3B6JSKR2U2RYNZLJPYW54Q4HXNGDCPBHUOQX5VNCWLDGCE35B4FQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.377657509 (531h44m26.248740544s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378008057 (531h44m26.249091092s) [noise=true , address=77.90.31.14:4661 , piece_id=OQTIZVTVXGONOJQ2XU2QKBGMKXQ6ZUATU7ANYOVKWLJIPHVZHBHQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378490385 (531h44m26.24957343s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378802850 (531h44m26.249885885s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.379142387 (531h44m26.250225422s) [noise=true , address=145.79.227.2:4113 , piece_id=MGBWZ7FDHLWM2TD3CDQ7VHUMLXUEOHWLR5P64FRAVE5QUK52VYDQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.379476383 (531h44m26.250559418s) [noise=true , address=77.90.31.10:4519 , piece_id=7YTYI2KH6D6Y2PGHROPP7OZATMF6QSL5QZCGIYXJOHWJBWM3DOYA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.380608759 (531h44m26.251691804s) [noise=true , address=145.79.227.48:4302 , piece_id=KU4RA7PQR3F5KJA55SZ2VIUDHPFOGX7XRECZJECIOEAWFLICP5MA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.380676458 (531h44m26.251759493s) [noise=true , address=45.91.116.13:4155 , piece_id=MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.381659790 (531h44m26.252742836s) [noise=true , address=145.79.227.9:4365 , piece_id=3FWWZUSNRBMFGHVALMQRAIFAANZT2QLZOIJVSGWBTOG3VL5CYGNQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.382353291 (531h44m26.253436336s) [noise=true , address=145.79.227.11:4732 , piece_id=AVL3Q7TMRHWGNZ22TYG5IPXIRMVEFJY4W3ULIUPUPI23NLFSSXGA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.383714052 (531h44m26.254797097s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.385082849 (531h44m26.256165894s) [noise=true , address=145.79.227.37:4565 , piece_id=HFM74QOLJTELRMBPFG4PWLXMQOBZD5SLSMQZCGOSTZXBBBKJHKGA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:28.089194807 (531h44m27.960277842s) [upload=466337]: Next returned nil segment Dec 27 19:13:28.089217240 (531h44m27.960300275s) [upload=466337]: Waiting for error group managing segments... Dec 27 19:13:28.387646493 (531h44m28.258729528s) [noise=true , address=145.79.227.48:4302 , piece_id=KU4RA7PQR3F5KJA55SZ2VIUDHPFOGX7XRECZJECIOEAWFLICP5MA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.388527130 (531h44m28.259610166s) [noise=true , address=145.79.227.37:4565 , piece_id=HFM74QOLJTELRMBPFG4PWLXMQOBZD5SLSMQZCGOSTZXBBBKJHKGA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.389251960 (531h44m28.260334995s) [noise=true , address=145.79.227.5:4223 , piece_id=T4TVGCCPE5BPIJ3K35QDTNNGMTQTJRHI7KSCNYC6B2DZSDAKDIHA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.395362295 (531h44m28.266445381s) [noise=true , address=145.79.227.2:4113 , piece_id=MGBWZ7FDHLWM2TD3CDQ7VHUMLXUEOHWLR5P64FRAVE5QUK52VYDQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.398843915 (531h44m28.26992696s) [noise=true , address=145.79.227.50:4374 , piece_id=3B6JSKR2U2RYNZLJPYW54Q4HXNGDCPBHUOQX5VNCWLDGCE35B4FQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.406970360 (531h44m28.278053395s) [noise=true , address=145.79.227.47:4265 , piece_id=WTSCDTBRSDO2Y3MQL4PZ4KP2PO4J3CEP2Z2LCWHYI7J7Y3D7LITQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.407352808 (531h44m28.278435853s) [noise=true , address=145.79.227.11:4732 , piece_id=AVL3Q7TMRHWGNZ22TYG5IPXIRMVEFJY4W3ULIUPUPI23NLFSSXGA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.416185658 (531h44m28.287268703s) [noise=true , address=145.79.227.9:4365 , piece_id=3FWWZUSNRBMFGHVALMQRAIFAANZT2QLZOIJVSGWBTOG3VL5CYGNQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.592079671 (531h44m28.463162706s) [noise=true , address=45.91.116.11:4117 , piece_id=NDAPJYPSODZPJHF4HIC5NAR4WV5USXKG3VHTKNRFNZPXUZ4SIDMA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.637960927 (531h44m28.509043972s) [noise=true , address=77.90.31.10:4519 , piece_id=7YTYI2KH6D6Y2PGHROPP7OZATMF6QSL5QZCGIYXJOHWJBWM3DOYA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.657276147 (531h44m28.528359182s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.681990348 (531h44m28.553073383s) [noise=true , address=77.90.31.14:4661 , piece_id=OQTIZVTVXGONOJQ2XU2QKBGMKXQ6ZUATU7ANYOVKWLJIPHVZHBHQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.734861374 (531h44m28.605944429s) [noise=true , address=45.91.116.13:4155 , piece_id=MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.734881822 (531h44m28.605964867s) [seg_pos={0 0}, upload=466337]: Segment reached optimal threshold of 13 pieces. Dec 27 19:13:28.735090229 (531h44m28.606173264s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection Dec 27 19:13:28.735097934 (531h44m28.606180969s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735215498 (531h44m28.606298533s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection Dec 27 19:13:28.735236849 (531h44m28.606319884s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735340145 (531h44m28.606423181s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 19:13:28.735347209 (531h44m28.606430244s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735365884 (531h44m28.606448919s) [seg_pos={0 0}, upload=466337]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:13:28.735450015 (531h44m28.60653305s) [upload=466337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:13:28.735456547 (531h44m28.606539582s) [upload=466337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:13:29.034869243 (531h44m28.905952278s) [upload=466337]: Done uploading segments... Dec 27 19:13:29.034889040 (531h44m28.905972085s) [upload=466337]: Upload finished. err: <nil> Duration: 13.839095999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXXX..XXX.XXXX........XX..XX..X.....X.........X....XXX..XX.] duration: 5.978229454s trace: https://jaeger.dev.storj.io/trace/3156bafcda88c4d5 FAIL (2025-12-27 19:14:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ) context deadline exceeded ======LOGS====== ng segments... Dec 27 19:13:26.098311945 (531h44m25.96939499s) [upload=466337]: Got next segment. Inline: false Dec 27 19:13:26.098363603 (531h44m25.969446638s) [upload=466337]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 19:13:26.098369896 (531h44m25.969452931s) [upload=466337]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 19:13:26.376014842 (531h44m26.247097877s) [seg_pos={0 0}, upload=466337]: Begin upload segment... Dec 27 19:13:26.376079475 (531h44m26.24716252s) [seg_pos={0 0}, upload=466337]: Done begin upload segment. Dec 27 19:13:26.376088552 (531h44m26.247171597s) [noise=true , address=145.79.227.5:4223 , piece_id=T4TVGCCPE5BPIJ3K35QDTNNGMTQTJRHI7KSCNYC6B2DZSDAKDIHA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.376114982 (531h44m26.247198038s) [noise=true , address=45.91.116.11:4117 , piece_id=NDAPJYPSODZPJHF4HIC5NAR4WV5USXKG3VHTKNRFNZPXUZ4SIDMA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.376932278 (531h44m26.248015313s) [noise=true , address=145.79.227.47:4265 , piece_id=WTSCDTBRSDO2Y3MQL4PZ4KP2PO4J3CEP2Z2LCWHYI7J7Y3D7LITQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.377325036 (531h44m26.248408071s) [noise=true , address=145.79.227.50:4374 , piece_id=3B6JSKR2U2RYNZLJPYW54Q4HXNGDCPBHUOQX5VNCWLDGCE35B4FQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.377657509 (531h44m26.248740544s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378008057 (531h44m26.249091092s) [noise=true , address=77.90.31.14:4661 , piece_id=OQTIZVTVXGONOJQ2XU2QKBGMKXQ6ZUATU7ANYOVKWLJIPHVZHBHQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378490385 (531h44m26.24957343s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.378802850 (531h44m26.249885885s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.379142387 (531h44m26.250225422s) [noise=true , address=145.79.227.2:4113 , piece_id=MGBWZ7FDHLWM2TD3CDQ7VHUMLXUEOHWLR5P64FRAVE5QUK52VYDQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.379476383 (531h44m26.250559418s) [noise=true , address=77.90.31.10:4519 , piece_id=7YTYI2KH6D6Y2PGHROPP7OZATMF6QSL5QZCGIYXJOHWJBWM3DOYA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.380608759 (531h44m26.251691804s) [noise=true , address=145.79.227.48:4302 , piece_id=KU4RA7PQR3F5KJA55SZ2VIUDHPFOGX7XRECZJECIOEAWFLICP5MA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.380676458 (531h44m26.251759493s) [noise=true , address=45.91.116.13:4155 , piece_id=MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.381659790 (531h44m26.252742836s) [noise=true , address=145.79.227.9:4365 , piece_id=3FWWZUSNRBMFGHVALMQRAIFAANZT2QLZOIJVSGWBTOG3VL5CYGNQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.382353291 (531h44m26.253436336s) [noise=true , address=145.79.227.11:4732 , piece_id=AVL3Q7TMRHWGNZ22TYG5IPXIRMVEFJY4W3ULIUPUPI23NLFSSXGA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.383714052 (531h44m26.254797097s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:26.385082849 (531h44m26.256165894s) [noise=true , address=145.79.227.37:4565 , piece_id=HFM74QOLJTELRMBPFG4PWLXMQOBZD5SLSMQZCGOSTZXBBBKJHKGA, seg_pos={0 0}, upload=466337]: Uploading piece... Dec 27 19:13:28.089194807 (531h44m27.960277842s) [upload=466337]: Next returned nil segment Dec 27 19:13:28.089217240 (531h44m27.960300275s) [upload=466337]: Waiting for error group managing segments... Dec 27 19:13:28.387646493 (531h44m28.258729528s) [noise=true , address=145.79.227.48:4302 , piece_id=KU4RA7PQR3F5KJA55SZ2VIUDHPFOGX7XRECZJECIOEAWFLICP5MA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.388527130 (531h44m28.259610166s) [noise=true , address=145.79.227.37:4565 , piece_id=HFM74QOLJTELRMBPFG4PWLXMQOBZD5SLSMQZCGOSTZXBBBKJHKGA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.389251960 (531h44m28.260334995s) [noise=true , address=145.79.227.5:4223 , piece_id=T4TVGCCPE5BPIJ3K35QDTNNGMTQTJRHI7KSCNYC6B2DZSDAKDIHA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.395362295 (531h44m28.266445381s) [noise=true , address=145.79.227.2:4113 , piece_id=MGBWZ7FDHLWM2TD3CDQ7VHUMLXUEOHWLR5P64FRAVE5QUK52VYDQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.398843915 (531h44m28.26992696s) [noise=true , address=145.79.227.50:4374 , piece_id=3B6JSKR2U2RYNZLJPYW54Q4HXNGDCPBHUOQX5VNCWLDGCE35B4FQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.406970360 (531h44m28.278053395s) [noise=true , address=145.79.227.47:4265 , piece_id=WTSCDTBRSDO2Y3MQL4PZ4KP2PO4J3CEP2Z2LCWHYI7J7Y3D7LITQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.407352808 (531h44m28.278435853s) [noise=true , address=145.79.227.11:4732 , piece_id=AVL3Q7TMRHWGNZ22TYG5IPXIRMVEFJY4W3ULIUPUPI23NLFSSXGA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.416185658 (531h44m28.287268703s) [noise=true , address=145.79.227.9:4365 , piece_id=3FWWZUSNRBMFGHVALMQRAIFAANZT2QLZOIJVSGWBTOG3VL5CYGNQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.592079671 (531h44m28.463162706s) [noise=true , address=45.91.116.11:4117 , piece_id=NDAPJYPSODZPJHF4HIC5NAR4WV5USXKG3VHTKNRFNZPXUZ4SIDMA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.637960927 (531h44m28.509043972s) [noise=true , address=77.90.31.10:4519 , piece_id=7YTYI2KH6D6Y2PGHROPP7OZATMF6QSL5QZCGIYXJOHWJBWM3DOYA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.657276147 (531h44m28.528359182s) [noise=true , address=45.91.116.10:4083 , piece_id=EBJSLUUKNTNINIHCXQSX7G4GSOT4G2YTMITOYNBBRUDD443S25HQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.681990348 (531h44m28.553073383s) [noise=true , address=77.90.31.14:4661 , piece_id=OQTIZVTVXGONOJQ2XU2QKBGMKXQ6ZUATU7ANYOVKWLJIPHVZHBHQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.734861374 (531h44m28.605944429s) [noise=true , address=45.91.116.13:4155 , piece_id=MJRM25OH3RTC3BGW6HGAUCIIQAGHJBMNRWG3M3AZZ2KG3EMDZ6HA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: <nil> Dec 27 19:13:28.734881822 (531h44m28.605964867s) [seg_pos={0 0}, upload=466337]: Segment reached optimal threshold of 13 pieces. Dec 27 19:13:28.735090229 (531h44m28.606173264s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49330->45.91.116.8:4045: use of closed network connection Dec 27 19:13:28.735097934 (531h44m28.606180969s) [noise=true , address=45.91.116.8:4045 , piece_id=JHKY34UBLXL5S63WMD3WLFF3STRQZHWSOTEMSLZKASJS2YSTRN5Q, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735215498 (531h44m28.606298533s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->45.91.116.5:4297: use of closed network connection Dec 27 19:13:28.735236849 (531h44m28.606319884s) [noise=true , address=45.91.116.5:4297 , piece_id=SWKCLCVYWO274NQ36UHALLYYW3BCZ3GCBGKB4XQPVPWURTMIG7NQ, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735340145 (531h44m28.606423181s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 27 19:13:28.735347209 (531h44m28.606430244s) [noise=true , address=77.90.31.12:4589 , piece_id=GAGJA2T73JOAZFT72QIZWWLIBKKASACVP2N73YEZ6MCEXA6TULLA, seg_pos={0 0}, upload=466337]: Long tail cancellation due to optimal threshold Dec 27 19:13:28.735365884 (531h44m28.606448919s) [seg_pos={0 0}, upload=466337]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 19:13:28.735450015 (531h44m28.60653305s) [upload=466337]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 19:13:28.735456547 (531h44m28.606539582s) [upload=466337]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 19:13:29.034869243 (531h44m28.905952278s) [upload=466337]: Done uploading segments... Dec 27 19:13:29.034889040 (531h44m28.905972085s) [upload=466337]: Upload finished. err: <nil> Duration: 5.978277424s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)