Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 869437 reports (Page 11614 of 17389)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 23.261781586s trace: https://jaeger.dev.storj.io/trace/1118adcb440fbf2c FAIL (2025-12-09 12:28:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:26:07.828800783 (92h57m7.699883828s) [upload=81055]: Starting upload Dec 9 12:26:07.828895262 (92h57m7.699978307s) [upload=81055]: Done starting upload Dec 9 12:26:07.828991666 (92h57m7.700074701s) [upload=81055]: Uploading segments... Dec 9 12:26:07.829245018 (92h57m7.700328063s) [upload=81055]: Got next segment. Inline: false Dec 9 12:26:07.829293751 (92h57m7.700376797s) [upload=81055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:26:07.829300795 (92h57m7.70038384s) [upload=81055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:26:08.182845342 (92h57m8.053928397s) [seg_pos={0 0}, upload=81055]: Begin upload segment... Dec 9 12:26:08.182984577 (92h57m8.054067623s) [seg_pos={0 0}, upload=81055]: Done begin upload segment. Dec 9 12:26:08.183027178 (92h57m8.054110223s) [noise=true , address=145.79.227.23:4546 , piece_id=BDXCJ3JQCHIFD3TJ36RVWQXT7DUMQ64ZIOSAPUV6TJKGAZG3ZVNQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.184550149 (92h57m8.055633194s) [noise=true , address=77.90.31.19:4842 , piece_id=BW2OYEWFJJG6SJNJFFSYKMU23IQPAWKEFCOXZFLUPYRDYFP54Q3Q, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185000606 (92h57m8.056083651s) [noise=true , address=77.90.31.18:4807 , piece_id=U3JW6G6EKFO4WMND3DSIIOQYSQAYOVNF65QVQBGDOEFPPHVQN7EQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185466894 (92h57m8.056549939s) [noise=true , address=145.79.227.40:4050 , piece_id=DVYH5MM3VA3FLFNYR3FDNJXOA6VG6EJWQ74TSLDIVX5N7SL4M4PQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185991603 (92h57m8.057074658s) [noise=true , address=145.79.227.42:4122 , piece_id=5XM2XG46J2WHSG2B5NE4QRR7ICYI2HYBHDIAU3DD73JZZ4YVVSVA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.186639487 (92h57m8.057722522s) [noise=true , address=145.79.227.29:4943 , piece_id=BKEW2PC5VA4JESBMGRHWMXE3BIMDTFVADT3PNXUC24BJOLWWDYWQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.187197359 (92h57m8.058280404s) [noise=true , address=145.79.227.39:4013 , piece_id=7LXP3X74TA52GE6DYJKOXPD4C5OQ33M3SSYZQWFNG5MSYXGYWFCQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.187659439 (92h57m8.058742484s) [noise=true , address=145.79.227.25:4617 , piece_id=DKKVJV4PIOCMTXWGMCXVOABS2ZB2Q3ITOSMLWNNXVBZ3B6BY3LCA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.189250880 (92h57m8.060333935s) [noise=true , address=145.79.227.6:4257 , piece_id=HLZG7W7RS6ZB65LKPIXXHX5MNY37IE4WZ73DRKIJJLY6IPNEWNBQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.191110120 (92h57m8.062193166s) [noise=true , address=77.90.31.14:4661 , piece_id=LCAXXXS4FWZOXI7PWBT4E5GYTPEURTVUKE63YSNXYFFXBMIRP7QA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.192801831 (92h57m8.063884876s) [noise=true , address=145.79.227.25:4618 , piece_id=W5VVYLDZU5HIAURHLNEPGPOBXGAP7USOEU6ERSW75UYZYFL5SNOQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.194967856 (92h57m8.066050891s) [noise=true , address=145.79.227.14:4838 , piece_id=AVBOJK65EO7SUDUYM36WQIW5H4TFK3FLPP6YOFLLK4QXJ2QZSC2Q, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.196838157 (92h57m8.067921202s) [noise=true , address=145.79.227.26:4654 , piece_id=RXK7QMEH62RMHSHC7ZUOTSCXYFFORLPTUDRK62LDATCRGG7KBHCA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.198996287 (92h57m8.070079332s) [noise=true , address=145.79.227.33:4871 , piece_id=V5TUFVTFBY4CEOT6YYC57LVACHR3MNCSTXMKDLGFRSHYZ7CUIEIA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.200338643 (92h57m8.071421688s) [noise=true , address=145.79.227.16:4912 , piece_id=MSX5ZEYUOF6XMQ5W5FWHKXND6LIEZNIV6IM6AFCN4MPUUAQO2ESA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.202258629 (92h57m8.073341664s) [noise=true , address=145.79.227.24:4581 , piece_id=DSCVFFPD2W2KWCBA77PV3P2MKD64QRPHXB6KGAK6N44WR5E57MYA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:09.863196890 (92h57m9.734279925s) [upload=81055]: Next returned nil segment Dec 9 12:26:09.863214343 (92h57m9.734297408s) [upload=81055]: Waiting for error group managing segments... Dec 9 12:26:10.267978323 (92h57m10.139061368s) [noise=true , address=77.90.31.14:4661 , piece_id=LCAXXXS4FWZOXI7PWBT4E5GYTPEURTVUKE63YSNXYFFXBMIRP7QA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:10.402490290 (92h57m10.273573335s) [noise=true , address=77.90.31.18:4807 , piece_id=U3JW6G6EKFO4WMND3DSIIOQYSQAYOVNF65QVQBGDOEFPPHVQN7EQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.113259804 (92h57m10.984342859s) [noise=true , address=145.79.227.29:4943 , piece_id=BKEW2PC5VA4JESBMGRHWMXE3BIMDTFVADT3PNXUC24BJOLWWDYWQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.128164735 (92h57m10.99924777s) [noise=true , address=145.79.227.24:4581 , piece_id=DSCVFFPD2W2KWCBA77PV3P2MKD64QRPHXB6KGAK6N44WR5E57MYA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.132658263 (92h57m11.003741298s) [noise=true , address=145.79.227.14:4838 , piece_id=AVBOJK65EO7SUDUYM36WQIW5H4TFK3FLPP6YOFLLK4QXJ2QZSC2Q, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.135828439 (92h57m11.006911484s) [noise=true , address=145.79.227.42:4122 , piece_id=5XM2XG46J2WHSG2B5NE4QRR7ICYI2HYBHDIAU3DD73JZZ4YVVSVA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.160990703 (92h57m11.032073748s) [noise=true , address=145.79.227.39:4013 , piece_id=7LXP3X74TA52GE6DYJKOXPD4C5OQ33M3SSYZQWFNG5MSYXGYWFCQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.176028798 (92h57m11.047111833s) [noise=true , address=145.79.227.33:4871 , piece_id=V5TUFVTFBY4CEOT6YYC57LVACHR3MNCSTXMKDLGFRSHYZ7CUIEIA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.182099107 (92h57m11.053182162s) [noise=true , address=145.79.227.16:4912 , piece_id=MSX5ZEYUOF6XMQ5W5FWHKXND6LIEZNIV6IM6AFCN4MPUUAQO2ESA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.201224887 (92h57m11.072307942s) [noise=true , address=145.79.227.6:4257 , piece_id=HLZG7W7RS6ZB65LKPIXXHX5MNY37IE4WZ73DRKIJJLY6IPNEWNBQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.266482299 (92h57m11.137565334s) [noise=true , address=145.79.227.26:4654 , piece_id=RXK7QMEH62RMHSHC7ZUOTSCXYFFORLPTUDRK62LDATCRGG7KBHCA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.338921247 (92h57m11.210004312s) [noise=true , address=145.79.227.40:4050 , piece_id=DVYH5MM3VA3FLFNYR3FDNJXOA6VG6EJWQ74TSLDIVX5N7SL4M4PQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.641763849 (92h57m11.512846894s) [noise=true , address=77.90.31.19:4842 , piece_id=BW2OYEWFJJG6SJNJFFSYKMU23IQPAWKEFCOXZFLUPYRDYFP54Q3Q, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 9 12:26:36.777632885 (92h57m36.64871592s) [noise=true , address=145.79.227.25:4618 , piece_id=W5VVYLDZU5HIAURHLNEPGPOBXGAP7USOEU6ERSW75UYZYFL5SNOQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection Dec 9 12:26:36.777877210 (92h57m36.648960256s) [noise=true , address=145.79.227.23:4546 , piece_id=BDXCJ3JQCHIFD3TJ36RVWQXT7DUMQ64ZIOSAPUV6TJKGAZG3ZVNQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection Dec 9 12:26:36.778088372 (92h57m36.649171418s) [noise=true , address=145.79.227.25:4617 , piece_id=DKKVJV4PIOCMTXWGMCXVOABS2ZB2Q3ITOSMLWNNXVBZ3B6BY3LCA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection Dec 9 12:26:36.778128549 (92h57m36.649211594s) [seg_pos={0 0}, upload=81055]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:26:37.126598605 (92h57m36.99768165s) [upload=81055]: Done uploading segments... Dec 9 12:26:37.126633782 (92h57m36.997716817s) [upload=81055]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 23.261810128s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.............................................X...X...X..XX..] duration: 10.012325154s trace: https://jaeger.dev.storj.io/trace/36cbf03830bdb66a FAIL (2025-12-09 12:28:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.012398862s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...XX................X.....X..........X.....X...X.......X...] duration: 3.432974403s trace: https://jaeger.dev.storj.io/trace/1d1adb4010e7ae02 FAIL (2025-12-09 12:28:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.433000061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 8.394508667s trace: https://jaeger.dev.storj.io/trace/1118adcb440fbf2c FAIL (2025-12-09 12:27:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:26:07.828800783 (92h57m7.699883828s) [upload=81055]: Starting upload Dec 9 12:26:07.828895262 (92h57m7.699978307s) [upload=81055]: Done starting upload Dec 9 12:26:07.828991666 (92h57m7.700074701s) [upload=81055]: Uploading segments... Dec 9 12:26:07.829245018 (92h57m7.700328063s) [upload=81055]: Got next segment. Inline: false Dec 9 12:26:07.829293751 (92h57m7.700376797s) [upload=81055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:26:07.829300795 (92h57m7.70038384s) [upload=81055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:26:08.182845342 (92h57m8.053928397s) [seg_pos={0 0}, upload=81055]: Begin upload segment... Dec 9 12:26:08.182984577 (92h57m8.054067623s) [seg_pos={0 0}, upload=81055]: Done begin upload segment. Dec 9 12:26:08.183027178 (92h57m8.054110223s) [noise=true , address=145.79.227.23:4546 , piece_id=BDXCJ3JQCHIFD3TJ36RVWQXT7DUMQ64ZIOSAPUV6TJKGAZG3ZVNQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.184550149 (92h57m8.055633194s) [noise=true , address=77.90.31.19:4842 , piece_id=BW2OYEWFJJG6SJNJFFSYKMU23IQPAWKEFCOXZFLUPYRDYFP54Q3Q, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185000606 (92h57m8.056083651s) [noise=true , address=77.90.31.18:4807 , piece_id=U3JW6G6EKFO4WMND3DSIIOQYSQAYOVNF65QVQBGDOEFPPHVQN7EQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185466894 (92h57m8.056549939s) [noise=true , address=145.79.227.40:4050 , piece_id=DVYH5MM3VA3FLFNYR3FDNJXOA6VG6EJWQ74TSLDIVX5N7SL4M4PQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185991603 (92h57m8.057074658s) [noise=true , address=145.79.227.42:4122 , piece_id=5XM2XG46J2WHSG2B5NE4QRR7ICYI2HYBHDIAU3DD73JZZ4YVVSVA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.186639487 (92h57m8.057722522s) [noise=true , address=145.79.227.29:4943 , piece_id=BKEW2PC5VA4JESBMGRHWMXE3BIMDTFVADT3PNXUC24BJOLWWDYWQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.187197359 (92h57m8.058280404s) [noise=true , address=145.79.227.39:4013 , piece_id=7LXP3X74TA52GE6DYJKOXPD4C5OQ33M3SSYZQWFNG5MSYXGYWFCQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.187659439 (92h57m8.058742484s) [noise=true , address=145.79.227.25:4617 , piece_id=DKKVJV4PIOCMTXWGMCXVOABS2ZB2Q3ITOSMLWNNXVBZ3B6BY3LCA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.189250880 (92h57m8.060333935s) [noise=true , address=145.79.227.6:4257 , piece_id=HLZG7W7RS6ZB65LKPIXXHX5MNY37IE4WZ73DRKIJJLY6IPNEWNBQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.191110120 (92h57m8.062193166s) [noise=true , address=77.90.31.14:4661 , piece_id=LCAXXXS4FWZOXI7PWBT4E5GYTPEURTVUKE63YSNXYFFXBMIRP7QA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.192801831 (92h57m8.063884876s) [noise=true , address=145.79.227.25:4618 , piece_id=W5VVYLDZU5HIAURHLNEPGPOBXGAP7USOEU6ERSW75UYZYFL5SNOQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.194967856 (92h57m8.066050891s) [noise=true , address=145.79.227.14:4838 , piece_id=AVBOJK65EO7SUDUYM36WQIW5H4TFK3FLPP6YOFLLK4QXJ2QZSC2Q, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.196838157 (92h57m8.067921202s) [noise=true , address=145.79.227.26:4654 , piece_id=RXK7QMEH62RMHSHC7ZUOTSCXYFFORLPTUDRK62LDATCRGG7KBHCA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.198996287 (92h57m8.070079332s) [noise=true , address=145.79.227.33:4871 , piece_id=V5TUFVTFBY4CEOT6YYC57LVACHR3MNCSTXMKDLGFRSHYZ7CUIEIA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.200338643 (92h57m8.071421688s) [noise=true , address=145.79.227.16:4912 , piece_id=MSX5ZEYUOF6XMQ5W5FWHKXND6LIEZNIV6IM6AFCN4MPUUAQO2ESA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.202258629 (92h57m8.073341664s) [noise=true , address=145.79.227.24:4581 , piece_id=DSCVFFPD2W2KWCBA77PV3P2MKD64QRPHXB6KGAK6N44WR5E57MYA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:09.863196890 (92h57m9.734279925s) [upload=81055]: Next returned nil segment Dec 9 12:26:09.863214343 (92h57m9.734297408s) [upload=81055]: Waiting for error group managing segments... Dec 9 12:26:10.267978323 (92h57m10.139061368s) [noise=true , address=77.90.31.14:4661 , piece_id=LCAXXXS4FWZOXI7PWBT4E5GYTPEURTVUKE63YSNXYFFXBMIRP7QA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:10.402490290 (92h57m10.273573335s) [noise=true , address=77.90.31.18:4807 , piece_id=U3JW6G6EKFO4WMND3DSIIOQYSQAYOVNF65QVQBGDOEFPPHVQN7EQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.113259804 (92h57m10.984342859s) [noise=true , address=145.79.227.29:4943 , piece_id=BKEW2PC5VA4JESBMGRHWMXE3BIMDTFVADT3PNXUC24BJOLWWDYWQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.128164735 (92h57m10.99924777s) [noise=true , address=145.79.227.24:4581 , piece_id=DSCVFFPD2W2KWCBA77PV3P2MKD64QRPHXB6KGAK6N44WR5E57MYA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.132658263 (92h57m11.003741298s) [noise=true , address=145.79.227.14:4838 , piece_id=AVBOJK65EO7SUDUYM36WQIW5H4TFK3FLPP6YOFLLK4QXJ2QZSC2Q, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.135828439 (92h57m11.006911484s) [noise=true , address=145.79.227.42:4122 , piece_id=5XM2XG46J2WHSG2B5NE4QRR7ICYI2HYBHDIAU3DD73JZZ4YVVSVA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.160990703 (92h57m11.032073748s) [noise=true , address=145.79.227.39:4013 , piece_id=7LXP3X74TA52GE6DYJKOXPD4C5OQ33M3SSYZQWFNG5MSYXGYWFCQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.176028798 (92h57m11.047111833s) [noise=true , address=145.79.227.33:4871 , piece_id=V5TUFVTFBY4CEOT6YYC57LVACHR3MNCSTXMKDLGFRSHYZ7CUIEIA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.182099107 (92h57m11.053182162s) [noise=true , address=145.79.227.16:4912 , piece_id=MSX5ZEYUOF6XMQ5W5FWHKXND6LIEZNIV6IM6AFCN4MPUUAQO2ESA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.201224887 (92h57m11.072307942s) [noise=true , address=145.79.227.6:4257 , piece_id=HLZG7W7RS6ZB65LKPIXXHX5MNY37IE4WZ73DRKIJJLY6IPNEWNBQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.266482299 (92h57m11.137565334s) [noise=true , address=145.79.227.26:4654 , piece_id=RXK7QMEH62RMHSHC7ZUOTSCXYFFORLPTUDRK62LDATCRGG7KBHCA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.338921247 (92h57m11.210004312s) [noise=true , address=145.79.227.40:4050 , piece_id=DVYH5MM3VA3FLFNYR3FDNJXOA6VG6EJWQ74TSLDIVX5N7SL4M4PQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.641763849 (92h57m11.512846894s) [noise=true , address=77.90.31.19:4842 , piece_id=BW2OYEWFJJG6SJNJFFSYKMU23IQPAWKEFCOXZFLUPYRDYFP54Q3Q, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 9 12:26:36.777632885 (92h57m36.64871592s) [noise=true , address=145.79.227.25:4618 , piece_id=W5VVYLDZU5HIAURHLNEPGPOBXGAP7USOEU6ERSW75UYZYFL5SNOQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection Dec 9 12:26:36.777877210 (92h57m36.648960256s) [noise=true , address=145.79.227.23:4546 , piece_id=BDXCJ3JQCHIFD3TJ36RVWQXT7DUMQ64ZIOSAPUV6TJKGAZG3ZVNQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection Dec 9 12:26:36.778088372 (92h57m36.649171418s) [noise=true , address=145.79.227.25:4617 , piece_id=DKKVJV4PIOCMTXWGMCXVOABS2ZB2Q3ITOSMLWNNXVBZ3B6BY3LCA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection Dec 9 12:26:36.778128549 (92h57m36.649211594s) [seg_pos={0 0}, upload=81055]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:26:37.126598605 (92h57m36.99768165s) [upload=81055]: Done uploading segments... Dec 9 12:26:37.126633782 (92h57m36.997716817s) [upload=81055]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.394552529s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....XX................X.....X..........X.....X...X.......X..] duration: 4.451970724s trace: https://jaeger.dev.storj.io/trace/1d1adb4010e7ae02 FAIL (2025-12-09 12:27:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.452002473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 16.344423869s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:27:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 16.344453956s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..............................................X...X...X..XX.] duration: 6.189541349s trace: https://jaeger.dev.storj.io/trace/36cbf03830bdb66a FAIL (2025-12-09 12:27:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.189581486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...............................................X...X...X..XX] duration: 1m0.001117558s trace: https://jaeger.dev.storj.io/trace/36cbf03830bdb66a FAIL (2025-12-09 12:27:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001144559s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....XX................X.....X..........X.....X...X.......X.] duration: 4.464418006s trace: https://jaeger.dev.storj.io/trace/1d1adb4010e7ae02 FAIL (2025-12-09 12:27:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.464443654s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 18.355329744s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:27:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 18.355362407s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 18.447930483s trace: https://jaeger.dev.storj.io/trace/1118adcb440fbf2c FAIL (2025-12-09 12:26:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:26:07.828800783 (92h57m7.699883828s) [upload=81055]: Starting upload Dec 9 12:26:07.828895262 (92h57m7.699978307s) [upload=81055]: Done starting upload Dec 9 12:26:07.828991666 (92h57m7.700074701s) [upload=81055]: Uploading segments... Dec 9 12:26:07.829245018 (92h57m7.700328063s) [upload=81055]: Got next segment. Inline: false Dec 9 12:26:07.829293751 (92h57m7.700376797s) [upload=81055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:26:07.829300795 (92h57m7.70038384s) [upload=81055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:26:08.182845342 (92h57m8.053928397s) [seg_pos={0 0}, upload=81055]: Begin upload segment... Dec 9 12:26:08.182984577 (92h57m8.054067623s) [seg_pos={0 0}, upload=81055]: Done begin upload segment. Dec 9 12:26:08.183027178 (92h57m8.054110223s) [noise=true , address=145.79.227.23:4546 , piece_id=BDXCJ3JQCHIFD3TJ36RVWQXT7DUMQ64ZIOSAPUV6TJKGAZG3ZVNQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.184550149 (92h57m8.055633194s) [noise=true , address=77.90.31.19:4842 , piece_id=BW2OYEWFJJG6SJNJFFSYKMU23IQPAWKEFCOXZFLUPYRDYFP54Q3Q, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185000606 (92h57m8.056083651s) [noise=true , address=77.90.31.18:4807 , piece_id=U3JW6G6EKFO4WMND3DSIIOQYSQAYOVNF65QVQBGDOEFPPHVQN7EQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185466894 (92h57m8.056549939s) [noise=true , address=145.79.227.40:4050 , piece_id=DVYH5MM3VA3FLFNYR3FDNJXOA6VG6EJWQ74TSLDIVX5N7SL4M4PQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185991603 (92h57m8.057074658s) [noise=true , address=145.79.227.42:4122 , piece_id=5XM2XG46J2WHSG2B5NE4QRR7ICYI2HYBHDIAU3DD73JZZ4YVVSVA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.186639487 (92h57m8.057722522s) [noise=true , address=145.79.227.29:4943 , piece_id=BKEW2PC5VA4JESBMGRHWMXE3BIMDTFVADT3PNXUC24BJOLWWDYWQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.187197359 (92h57m8.058280404s) [noise=true , address=145.79.227.39:4013 , piece_id=7LXP3X74TA52GE6DYJKOXPD4C5OQ33M3SSYZQWFNG5MSYXGYWFCQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.187659439 (92h57m8.058742484s) [noise=true , address=145.79.227.25:4617 , piece_id=DKKVJV4PIOCMTXWGMCXVOABS2ZB2Q3ITOSMLWNNXVBZ3B6BY3LCA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.189250880 (92h57m8.060333935s) [noise=true , address=145.79.227.6:4257 , piece_id=HLZG7W7RS6ZB65LKPIXXHX5MNY37IE4WZ73DRKIJJLY6IPNEWNBQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.191110120 (92h57m8.062193166s) [noise=true , address=77.90.31.14:4661 , piece_id=LCAXXXS4FWZOXI7PWBT4E5GYTPEURTVUKE63YSNXYFFXBMIRP7QA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.192801831 (92h57m8.063884876s) [noise=true , address=145.79.227.25:4618 , piece_id=W5VVYLDZU5HIAURHLNEPGPOBXGAP7USOEU6ERSW75UYZYFL5SNOQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.194967856 (92h57m8.066050891s) [noise=true , address=145.79.227.14:4838 , piece_id=AVBOJK65EO7SUDUYM36WQIW5H4TFK3FLPP6YOFLLK4QXJ2QZSC2Q, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.196838157 (92h57m8.067921202s) [noise=true , address=145.79.227.26:4654 , piece_id=RXK7QMEH62RMHSHC7ZUOTSCXYFFORLPTUDRK62LDATCRGG7KBHCA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.198996287 (92h57m8.070079332s) [noise=true , address=145.79.227.33:4871 , piece_id=V5TUFVTFBY4CEOT6YYC57LVACHR3MNCSTXMKDLGFRSHYZ7CUIEIA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.200338643 (92h57m8.071421688s) [noise=true , address=145.79.227.16:4912 , piece_id=MSX5ZEYUOF6XMQ5W5FWHKXND6LIEZNIV6IM6AFCN4MPUUAQO2ESA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.202258629 (92h57m8.073341664s) [noise=true , address=145.79.227.24:4581 , piece_id=DSCVFFPD2W2KWCBA77PV3P2MKD64QRPHXB6KGAK6N44WR5E57MYA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:09.863196890 (92h57m9.734279925s) [upload=81055]: Next returned nil segment Dec 9 12:26:09.863214343 (92h57m9.734297408s) [upload=81055]: Waiting for error group managing segments... Dec 9 12:26:10.267978323 (92h57m10.139061368s) [noise=true , address=77.90.31.14:4661 , piece_id=LCAXXXS4FWZOXI7PWBT4E5GYTPEURTVUKE63YSNXYFFXBMIRP7QA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:10.402490290 (92h57m10.273573335s) [noise=true , address=77.90.31.18:4807 , piece_id=U3JW6G6EKFO4WMND3DSIIOQYSQAYOVNF65QVQBGDOEFPPHVQN7EQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.113259804 (92h57m10.984342859s) [noise=true , address=145.79.227.29:4943 , piece_id=BKEW2PC5VA4JESBMGRHWMXE3BIMDTFVADT3PNXUC24BJOLWWDYWQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.128164735 (92h57m10.99924777s) [noise=true , address=145.79.227.24:4581 , piece_id=DSCVFFPD2W2KWCBA77PV3P2MKD64QRPHXB6KGAK6N44WR5E57MYA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.132658263 (92h57m11.003741298s) [noise=true , address=145.79.227.14:4838 , piece_id=AVBOJK65EO7SUDUYM36WQIW5H4TFK3FLPP6YOFLLK4QXJ2QZSC2Q, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.135828439 (92h57m11.006911484s) [noise=true , address=145.79.227.42:4122 , piece_id=5XM2XG46J2WHSG2B5NE4QRR7ICYI2HYBHDIAU3DD73JZZ4YVVSVA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.160990703 (92h57m11.032073748s) [noise=true , address=145.79.227.39:4013 , piece_id=7LXP3X74TA52GE6DYJKOXPD4C5OQ33M3SSYZQWFNG5MSYXGYWFCQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.176028798 (92h57m11.047111833s) [noise=true , address=145.79.227.33:4871 , piece_id=V5TUFVTFBY4CEOT6YYC57LVACHR3MNCSTXMKDLGFRSHYZ7CUIEIA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.182099107 (92h57m11.053182162s) [noise=true , address=145.79.227.16:4912 , piece_id=MSX5ZEYUOF6XMQ5W5FWHKXND6LIEZNIV6IM6AFCN4MPUUAQO2ESA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.201224887 (92h57m11.072307942s) [noise=true , address=145.79.227.6:4257 , piece_id=HLZG7W7RS6ZB65LKPIXXHX5MNY37IE4WZ73DRKIJJLY6IPNEWNBQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.266482299 (92h57m11.137565334s) [noise=true , address=145.79.227.26:4654 , piece_id=RXK7QMEH62RMHSHC7ZUOTSCXYFFORLPTUDRK62LDATCRGG7KBHCA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.338921247 (92h57m11.210004312s) [noise=true , address=145.79.227.40:4050 , piece_id=DVYH5MM3VA3FLFNYR3FDNJXOA6VG6EJWQ74TSLDIVX5N7SL4M4PQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.641763849 (92h57m11.512846894s) [noise=true , address=77.90.31.19:4842 , piece_id=BW2OYEWFJJG6SJNJFFSYKMU23IQPAWKEFCOXZFLUPYRDYFP54Q3Q, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 9 12:26:36.777632885 (92h57m36.64871592s) [noise=true , address=145.79.227.25:4618 , piece_id=W5VVYLDZU5HIAURHLNEPGPOBXGAP7USOEU6ERSW75UYZYFL5SNOQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection Dec 9 12:26:36.777877210 (92h57m36.648960256s) [noise=true , address=145.79.227.23:4546 , piece_id=BDXCJ3JQCHIFD3TJ36RVWQXT7DUMQ64ZIOSAPUV6TJKGAZG3ZVNQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection Dec 9 12:26:36.778088372 (92h57m36.649171418s) [noise=true , address=145.79.227.25:4617 , piece_id=DKKVJV4PIOCMTXWGMCXVOABS2ZB2Q3ITOSMLWNNXVBZ3B6BY3LCA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection Dec 9 12:26:36.778128549 (92h57m36.649211594s) [seg_pos={0 0}, upload=81055]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:26:37.126598605 (92h57m36.99768165s) [upload=81055]: Done uploading segments... Dec 9 12:26:37.126633782 (92h57m36.997716817s) [upload=81055]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 18.447947966s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.350898401s trace: https://jaeger.dev.storj.io/trace/1118adcb440fbf2c FAIL (2025-12-09 12:26:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 9 12:26:07.828800783 (92h57m7.699883828s) [upload=81055]: Starting upload Dec 9 12:26:07.828895262 (92h57m7.699978307s) [upload=81055]: Done starting upload Dec 9 12:26:07.828991666 (92h57m7.700074701s) [upload=81055]: Uploading segments... Dec 9 12:26:07.829245018 (92h57m7.700328063s) [upload=81055]: Got next segment. Inline: false Dec 9 12:26:07.829293751 (92h57m7.700376797s) [upload=81055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 12:26:07.829300795 (92h57m7.70038384s) [upload=81055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 12:26:08.182845342 (92h57m8.053928397s) [seg_pos={0 0}, upload=81055]: Begin upload segment... Dec 9 12:26:08.182984577 (92h57m8.054067623s) [seg_pos={0 0}, upload=81055]: Done begin upload segment. Dec 9 12:26:08.183027178 (92h57m8.054110223s) [noise=true , address=145.79.227.23:4546 , piece_id=BDXCJ3JQCHIFD3TJ36RVWQXT7DUMQ64ZIOSAPUV6TJKGAZG3ZVNQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.184550149 (92h57m8.055633194s) [noise=true , address=77.90.31.19:4842 , piece_id=BW2OYEWFJJG6SJNJFFSYKMU23IQPAWKEFCOXZFLUPYRDYFP54Q3Q, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185000606 (92h57m8.056083651s) [noise=true , address=77.90.31.18:4807 , piece_id=U3JW6G6EKFO4WMND3DSIIOQYSQAYOVNF65QVQBGDOEFPPHVQN7EQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185466894 (92h57m8.056549939s) [noise=true , address=145.79.227.40:4050 , piece_id=DVYH5MM3VA3FLFNYR3FDNJXOA6VG6EJWQ74TSLDIVX5N7SL4M4PQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.185991603 (92h57m8.057074658s) [noise=true , address=145.79.227.42:4122 , piece_id=5XM2XG46J2WHSG2B5NE4QRR7ICYI2HYBHDIAU3DD73JZZ4YVVSVA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.186639487 (92h57m8.057722522s) [noise=true , address=145.79.227.29:4943 , piece_id=BKEW2PC5VA4JESBMGRHWMXE3BIMDTFVADT3PNXUC24BJOLWWDYWQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.187197359 (92h57m8.058280404s) [noise=true , address=145.79.227.39:4013 , piece_id=7LXP3X74TA52GE6DYJKOXPD4C5OQ33M3SSYZQWFNG5MSYXGYWFCQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.187659439 (92h57m8.058742484s) [noise=true , address=145.79.227.25:4617 , piece_id=DKKVJV4PIOCMTXWGMCXVOABS2ZB2Q3ITOSMLWNNXVBZ3B6BY3LCA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.189250880 (92h57m8.060333935s) [noise=true , address=145.79.227.6:4257 , piece_id=HLZG7W7RS6ZB65LKPIXXHX5MNY37IE4WZ73DRKIJJLY6IPNEWNBQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.191110120 (92h57m8.062193166s) [noise=true , address=77.90.31.14:4661 , piece_id=LCAXXXS4FWZOXI7PWBT4E5GYTPEURTVUKE63YSNXYFFXBMIRP7QA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.192801831 (92h57m8.063884876s) [noise=true , address=145.79.227.25:4618 , piece_id=W5VVYLDZU5HIAURHLNEPGPOBXGAP7USOEU6ERSW75UYZYFL5SNOQ, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.194967856 (92h57m8.066050891s) [noise=true , address=145.79.227.14:4838 , piece_id=AVBOJK65EO7SUDUYM36WQIW5H4TFK3FLPP6YOFLLK4QXJ2QZSC2Q, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.196838157 (92h57m8.067921202s) [noise=true , address=145.79.227.26:4654 , piece_id=RXK7QMEH62RMHSHC7ZUOTSCXYFFORLPTUDRK62LDATCRGG7KBHCA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.198996287 (92h57m8.070079332s) [noise=true , address=145.79.227.33:4871 , piece_id=V5TUFVTFBY4CEOT6YYC57LVACHR3MNCSTXMKDLGFRSHYZ7CUIEIA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.200338643 (92h57m8.071421688s) [noise=true , address=145.79.227.16:4912 , piece_id=MSX5ZEYUOF6XMQ5W5FWHKXND6LIEZNIV6IM6AFCN4MPUUAQO2ESA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:08.202258629 (92h57m8.073341664s) [noise=true , address=145.79.227.24:4581 , piece_id=DSCVFFPD2W2KWCBA77PV3P2MKD64QRPHXB6KGAK6N44WR5E57MYA, seg_pos={0 0}, upload=81055]: Uploading piece... Dec 9 12:26:09.863196890 (92h57m9.734279925s) [upload=81055]: Next returned nil segment Dec 9 12:26:09.863214343 (92h57m9.734297408s) [upload=81055]: Waiting for error group managing segments... Dec 9 12:26:10.267978323 (92h57m10.139061368s) [noise=true , address=77.90.31.14:4661 , piece_id=LCAXXXS4FWZOXI7PWBT4E5GYTPEURTVUKE63YSNXYFFXBMIRP7QA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:10.402490290 (92h57m10.273573335s) [noise=true , address=77.90.31.18:4807 , piece_id=U3JW6G6EKFO4WMND3DSIIOQYSQAYOVNF65QVQBGDOEFPPHVQN7EQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.113259804 (92h57m10.984342859s) [noise=true , address=145.79.227.29:4943 , piece_id=BKEW2PC5VA4JESBMGRHWMXE3BIMDTFVADT3PNXUC24BJOLWWDYWQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.128164735 (92h57m10.99924777s) [noise=true , address=145.79.227.24:4581 , piece_id=DSCVFFPD2W2KWCBA77PV3P2MKD64QRPHXB6KGAK6N44WR5E57MYA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.132658263 (92h57m11.003741298s) [noise=true , address=145.79.227.14:4838 , piece_id=AVBOJK65EO7SUDUYM36WQIW5H4TFK3FLPP6YOFLLK4QXJ2QZSC2Q, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.135828439 (92h57m11.006911484s) [noise=true , address=145.79.227.42:4122 , piece_id=5XM2XG46J2WHSG2B5NE4QRR7ICYI2HYBHDIAU3DD73JZZ4YVVSVA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.160990703 (92h57m11.032073748s) [noise=true , address=145.79.227.39:4013 , piece_id=7LXP3X74TA52GE6DYJKOXPD4C5OQ33M3SSYZQWFNG5MSYXGYWFCQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.176028798 (92h57m11.047111833s) [noise=true , address=145.79.227.33:4871 , piece_id=V5TUFVTFBY4CEOT6YYC57LVACHR3MNCSTXMKDLGFRSHYZ7CUIEIA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.182099107 (92h57m11.053182162s) [noise=true , address=145.79.227.16:4912 , piece_id=MSX5ZEYUOF6XMQ5W5FWHKXND6LIEZNIV6IM6AFCN4MPUUAQO2ESA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.201224887 (92h57m11.072307942s) [noise=true , address=145.79.227.6:4257 , piece_id=HLZG7W7RS6ZB65LKPIXXHX5MNY37IE4WZ73DRKIJJLY6IPNEWNBQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.266482299 (92h57m11.137565334s) [noise=true , address=145.79.227.26:4654 , piece_id=RXK7QMEH62RMHSHC7ZUOTSCXYFFORLPTUDRK62LDATCRGG7KBHCA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.338921247 (92h57m11.210004312s) [noise=true , address=145.79.227.40:4050 , piece_id=DVYH5MM3VA3FLFNYR3FDNJXOA6VG6EJWQ74TSLDIVX5N7SL4M4PQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: <nil> Dec 9 12:26:11.641763849 (92h57m11.512846894s) [noise=true , address=77.90.31.19:4842 , piece_id=BW2OYEWFJJG6SJNJFFSYKMU23IQPAWKEFCOXZFLUPYRDYFP54Q3Q, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: storage node overloaded, request limit: 2000 Dec 9 12:26:36.777632885 (92h57m36.64871592s) [noise=true , address=145.79.227.25:4618 , piece_id=W5VVYLDZU5HIAURHLNEPGPOBXGAP7USOEU6ERSW75UYZYFL5SNOQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8, address:145.79.227.25:4618): protocol: write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection; write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41782->145.79.227.25:4618: use of closed network connection Dec 9 12:26:36.777877210 (92h57m36.648960256s) [noise=true , address=145.79.227.23:4546 , piece_id=BDXCJ3JQCHIFD3TJ36RVWQXT7DUMQ64ZIOSAPUV6TJKGAZG3ZVNQ, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50454->145.79.227.23:4546: use of closed network connection Dec 9 12:26:36.778088372 (92h57m36.649171418s) [noise=true , address=145.79.227.25:4617 , piece_id=DKKVJV4PIOCMTXWGMCXVOABS2ZB2Q3ITOSMLWNNXVBZ3B6BY3LCA, seg_pos={0 0}, upload=81055]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59906->145.79.227.25:4617: use of closed network connection Dec 9 12:26:36.778128549 (92h57m36.649211594s) [seg_pos={0 0}, upload=81055]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 9 12:26:37.126598605 (92h57m36.99768165s) [upload=81055]: Done uploading segments... Dec 9 12:26:37.126633782 (92h57m36.997716817s) [upload=81055]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.350911977s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 4.658121263s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:26:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.658144077s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 21.174262053s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:26:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 21.174289715s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......XX................X.....X..........X.....X...X.......X] duration: 5.001174051s trace: https://jaeger.dev.storj.io/trace/1d1adb4010e7ae02 FAIL (2025-12-09 12:26:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001199639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 16.141100214s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:25:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 16.14112417s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [................................................X...X...X..X] duration: 1m0.001372225s trace: https://jaeger.dev.storj.io/trace/266770fd96e01ba2 FAIL (2025-12-09 12:25:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001396771s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 19.937806888s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:25:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 19.937833357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 14.2008168s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:25:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 14.200918864s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 14.003603549s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:25:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 14.003633876s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.......XX................X.....X..........X.....X...X.......] duration: 3.566169944s trace: https://jaeger.dev.storj.io/trace/27549a42879cb6b7 FAIL (2025-12-09 12:25:11 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566208348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 15.077557766s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:25:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 15.077587403s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 19.965248702s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:24:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 19.965276233s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X...X...X..] duration: 7.0764176s trace: https://jaeger.dev.storj.io/trace/60a0f3248a79c9f6 FAIL (2025-12-09 12:24:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.076442617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 18.825991456s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:24:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 18.826022466s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [........XX................X.....X..........X.....X...X......] duration: 4.858125291s trace: https://jaeger.dev.storj.io/trace/27549a42879cb6b7 FAIL (2025-12-09 12:24:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.858157893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 10.753215945s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:23:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 10.753246002s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 16.484715322s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:23:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 16.484741542s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.........XX................X.....X..........X.....X...X.....] duration: 4.846878264s trace: https://jaeger.dev.storj.io/trace/27549a42879cb6b7 FAIL (2025-12-09 12:23:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.846929271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X...X...X.] duration: 7.016593126s trace: https://jaeger.dev.storj.io/trace/60a0f3248a79c9f6 FAIL (2025-12-09 12:23:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.016620586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X...X...X] duration: 1m0.001057977s trace: https://jaeger.dev.storj.io/trace/60a0f3248a79c9f6 FAIL (2025-12-09 12:23:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001081891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 15.267067082s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:22:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 15.26709793s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 18.9445826s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:22:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 18.944598161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 19.859014982s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:22:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 19.859039358s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m12.868144458s trace: https://jaeger.dev.storj.io/trace/65adbc6b210718e5 FAIL (2025-12-09 12:22:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2m11.707458594s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.836921137 (92h52m11.708004182s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.838570889 (92h52m11.709653934s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839153668 (92h52m11.710236713s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.839694208 (92h52m11.710777253s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.840962833 (92h52m11.712045868s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.842244033 (92h52m11.713327078s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.843403420 (92h52m11.714486455s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.844696684 (92h52m11.715779719s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.845952355 (92h52m11.71703539s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847443374 (92h52m11.718526419s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.847781507 (92h52m11.718864542s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.849955197 (92h52m11.721038232s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.850557062 (92h52m11.721640098s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:11.851536218 (92h52m11.722619253s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Uploading piece... Dec 9 12:21:13.286101691 (92h52m13.157184726s) [upload=80984]: Next returned nil segment Dec 9 12:21:13.286123523 (92h52m13.157206568s) [upload=80984]: Waiting for error group managing segments... Dec 9 12:21:13.655440854 (92h52m13.526523899s) [noise=true , address=5.231.73.2:4001 , piece_id=UG6KETTLKXE2V4RL6QG3RHR67I2TBA4B52B5CQ2WNJWCOSY7O6BQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.780009016 (92h52m13.651092061s) [noise=true , address=77.90.31.19:4848 , piece_id=2VNEHYCZTLMOPB3FLLZH6ZLKWPBBTWNR5C36CNQGHGS6JS4EPC5Q, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.835658153 (92h52m13.706741198s) [noise=true , address=77.90.31.21:4885 , piece_id=SI7RKBGVJYPG7LWJUHFE66VMWWDQIBXBUOKGP2WHDWDYXM3AZ7CA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.843408692 (92h52m13.714491727s) [noise=true , address=77.90.31.12:4604 , piece_id=BCMJC4ZWT5JB3E4PFRBDOT44UN72DOV7GPCWWNU2MKMTHLPY2EYQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.891849994 (92h52m13.762933039s) [noise=true , address=77.90.31.13:4658 , piece_id=AUNKWR2HRXJLFK3ZJ6RKJ72GW7X27655XIF6UAOQXRFYMA3PZYGQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.893674758 (92h52m13.764757803s) [noise=true , address=77.90.31.18:4833 , piece_id=WZ6RFSZPZRFWNFIWGEKQWTGERPIKI27MI3VZUCFFNSS7ETT4JJAA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:13.896227378 (92h52m13.767310443s) [noise=true , address=77.90.31.17:4804 , piece_id=LMA26I4OANWJQBSHY2A7ZTLE5LDDTRGDD33PXQGXLRBHEED7BDXA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.343698982 (92h52m14.214782027s) [noise=true , address=5.231.73.5:4003 , piece_id=52FUKRVHVCSGHSTCSNGTOWVXIAE2SGBMVFMKH4EN2QTG3LIH3UMQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.455409521 (92h52m14.326492566s) [noise=true , address=145.79.227.27:4702 , piece_id=MTES57RWW7SIDNMTH3CMOFJPPGQDQK54EHFVQKQDDTPSBBCCACLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.472758147 (92h52m14.343841192s) [noise=true , address=145.79.227.44:4225 , piece_id=TWKNLNN4YCFLUVEAXKLWUHWHIWC37SGSS4HC646E3V5CK3M5Q5QQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:14.997763339 (92h52m14.868846374s) [noise=true , address=77.90.31.15:4728 , piece_id=K6UEBKWOKJ34HZS3E3NGDYN2JC3GDQUO5TM2W3VZNENJL2BL3BAQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:15.315485844 (92h52m15.186568879s) [noise=true , address=77.90.31.16:4765 , piece_id=QNLHL5L7CDKQQMTXKCF5V72SA4MUOLQJ7VPCMSL2F4O5434TICLQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:22.901605029 (92h52m22.772688074s) [noise=true , address=45.91.116.10:4110 , piece_id=GO7W6WNU2HMTU2LVP73WD24WNRQLI5TEIKF4N6XUUOO55CNPDZFA, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592471529 (92h52m24.463554574s) [noise=true , address=45.91.116.5:4323 , piece_id=QEG5BHSM76PAOMJ6XSEEFUEYV7A3AOKUCDYMILQP3HLUUF4IF37A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: <nil> Dec 9 12:21:24.592494553 (92h52m24.463577598s) [seg_pos={0 0}, upload=80984]: Segment reached optimal threshold of 14 pieces. Dec 9 12:21:24.592654457 (92h52m24.463737502s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.592661992 (92h52m24.463745037s) [noise=true , address=45.91.116.12:4256 , piece_id=K7HXMO7S5BS5GIOJYRA2RHDBP6VJWYMEZRNPYGNP3QGBSERP32ZQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.592933628 (92h52m24.464016663s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41910->64.49.13.169:4711: use of closed network connection Dec 9 12:21:24.592940773 (92h52m24.464023808s) [noise=true , address=64.49.13.169:4711 , piece_id=K27XD6QTCDZ4P7XVU7YQUX4DPFEAV2NOKZQVMZJLCLLXPYHC5YFQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593052866 (92h52m24.464135901s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UtzGEdmMaJAFkfprwLpTuxP2XDFtiN3Tgcqpnf9UtHVQzdhEg): protocol: expected piece hash; context canceled; EOF Dec 9 12:21:24.593059258 (92h52m24.464142303s) [noise=true , address=45.91.116.7:4035 , piece_id=CT5BQMGNZHEFMNJ6FI53CCC7UOY2VIEFFS42YQ6QFWEQY7C7CUVQ, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593152897 (92h52m24.464235942s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa): protocol: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48878->45.91.116.11:4132: use of closed network connection Dec 9 12:21:24.593164168 (92h52m24.464247213s) [noise=true , address=45.91.116.11:4132 , piece_id=BEXA3WIPV6BYIDCGWGQXHZDSGU6Y5FSAW4CUNHRKZ2AASKEJD52A, seg_pos={0 0}, upload=80984]: Long tail cancellation due to optimal threshold Dec 9 12:21:24.593184226 (92h52m24.464267261s) [seg_pos={0 0}, upload=80984]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 12:21:24.593265250 (92h52m24.464348295s) [upload=80984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 12:21:24.593272764 (92h52m24.46435581s) [upload=80984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 12:22:22.475738370 (92h53m22.346821415s) [upload=80984]: Done uploading segments... Dec 9 12:22:22.475787112 (92h53m22.346870157s) [upload=80984]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m12.868155289s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..........XX................X.....X..........X.....X...X....] duration: 3.765354241s trace: https://jaeger.dev.storj.io/trace/27549a42879cb6b7 FAIL (2025-12-09 12:22:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.765443932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 12.282476707s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:22:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 12.282502194s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X...X...] duration: 6.465164677s trace: https://jaeger.dev.storj.io/trace/1947f0931d35a9e7 FAIL (2025-12-09 12:21:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.465190647s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...........XX................X.....X..........X.....X...X...] duration: 3.53337203s trace: https://jaeger.dev.storj.io/trace/27549a42879cb6b7 FAIL (2025-12-09 12:21:09 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533400033s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 14.156551905s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:20:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 14.156704425s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [............XX................X.....X..........X.....X...X..] duration: 4.305938426s trace: https://jaeger.dev.storj.io/trace/27549a42879cb6b7 FAIL (2025-12-09 12:20:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.305970668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.............XX................X.....X..........X.....X...X.] duration: 3.512828252s trace: https://jaeger.dev.storj.io/trace/27549a42879cb6b7 FAIL (2025-12-09 12:20:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.512883728s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X...X..] duration: 5.789503572s trace: https://jaeger.dev.storj.io/trace/1947f0931d35a9e7 FAIL (2025-12-09 12:20:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.789532037s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [..............XX................X.....X..........X.....X...X] duration: 5.013309356s trace: https://jaeger.dev.storj.io/trace/27549a42879cb6b7 FAIL (2025-12-09 12:20:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.013349162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 15.028204999s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:19:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 15.028234544s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X...X.] duration: 6.832482822s trace: https://jaeger.dev.storj.io/trace/1947f0931d35a9e7 FAIL (2025-12-09 12:19:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.832501057s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.15967047s trace: https://jaeger.dev.storj.io/trace/7098c007ca020436 FAIL (2025-12-09 12:19:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362641042 (92h9m49.233724077s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.362747554 (92h9m49.233830599s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363050582 (92h9m49.234133627s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363113251 (92h9m49.234196286s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363383676 (92h9m49.234466711s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.363681573 (92h9m49.234764608s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364037821 (92h9m49.235120866s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364321191 (92h9m49.235404226s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.364596545 (92h9m49.23567959s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:49.365194575 (92h9m49.23627762s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Uploading piece... Dec 9 11:38:50.188365108 (92h9m50.059448153s) [upload=80368]: Next returned nil segment Dec 9 11:38:50.188379346 (92h9m50.059462391s) [upload=80368]: Waiting for error group managing segments... Dec 9 11:38:50.573198910 (92h9m50.444281955s) [noise=true , address=185.167.204.206:4062 , piece_id=UZUYTLORRUJOMEX62Z7ABCOXWZHBZ7PV72MA4TZIWDDWZKBR22OA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577545046 (92h9m50.448628081s) [noise=true , address=185.167.204.206:4059 , piece_id=HLZEIABVICZTZCKXHBKIFQJ3UVRGVV5EWHWFFL5G3E7CYGJN4GHA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.577840359 (92h9m50.448923394s) [noise=true , address=185.167.204.206:4045 , piece_id=7RI3JKTQDSVERWTVQ4NIGZMBNAC6OPYUCTPU77HX3V74NBG7XALQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.578430443 (92h9m50.449513478s) [noise=true , address=185.167.204.206:4048 , piece_id=XMV7K5EEISKMT5JQFUD6EXZ3EYELHBWF6Y3J276VUUP2YGXJKF3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.579912414 (92h9m50.450995459s) [noise=true , address=185.167.204.205:4016 , piece_id=55J2Z5J7LPDPCBKXAQ7V4BQW4U7W6UAQVF63LYH3BPFHSEQE6CEA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.580088579 (92h9m50.451171625s) [noise=true , address=185.167.204.206:4068 , piece_id=FDF7EQNPFGNEDO6CEPYGMAAJOIT4TPHY2PVYHNXH7ILC7AY2LO3A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.582475435 (92h9m50.45355848s) [noise=true , address=185.167.204.206:4037 , piece_id=LO332KWEY4NLBKVGEVME2SF7UIGTEVCKLOHRCC7FLIJEPP5DTKTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586259020 (92h9m50.457342055s) [noise=true , address=185.167.204.205:4030 , piece_id=VHBCZ6QQRBZ4THX6XCFIZ6JSFMIONQFSEZ3ERGOZB6L7L5USAKMA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.586458941 (92h9m50.457541986s) [noise=true , address=185.167.204.206:4060 , piece_id=A3NTK6DGBBOACMUONBGII27GVS7QZWOGDEKBPZFX76RSH5HMXS5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.588436907 (92h9m50.459519942s) [noise=true , address=185.167.204.205:4009 , piece_id=5WUDBEGUENMGEMSND2VCENBNGG2BIV47ICSWN3CJYLQQ3XIVZXKQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.589577038 (92h9m50.460660083s) [noise=true , address=185.167.204.205:4029 , piece_id=WMBEVUF62ZU76TQDICPDPMG4JA7374YQFXTI5XJFSN6MJZBATS7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:50.592840462 (92h9m50.463923507s) [noise=true , address=185.167.204.205:4001 , piece_id=KIUIPM26OHXISSEHSZPLFHUXF4NVOHSY6TILREFVGDRDBB5IOCQQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.141501915 (92h9m51.01258496s) [noise=true , address=81.171.14.70:4039 , piece_id=EJYQEP7LA6QJQICJQQVTNIXDN6ID5PAPL54GADSICUBJ3YQQ5E7Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153777871 (92h9m51.024860916s) [noise=true , address=81.171.14.70:4042 , piece_id=WWJDEHBRMRVXANEWVG6B6QLD2SR7M6FXH4642UZ3AFQSDGCO2HYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.153948696 (92h9m51.025031751s) [noise=true , address=81.171.14.70:4028 , piece_id=NSYGSKJBIHMHZZX7HK7N3UQDGKWWQARJPGLAWH2TEPGWKNYWPC4A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.157888027 (92h9m51.028971062s) [noise=true , address=81.171.14.70:4041 , piece_id=PZNQJLGVYH3CBPX4L2URCEVLMGK4JLQAMVFX25UBONKVYSTSEEWQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.170799434 (92h9m51.041882489s) [noise=true , address=81.171.14.70:4045 , piece_id=TZIZNHRR66JMWMZJNUHYFPYJ2ORR3OLKZ7VOQ5MIWXQOFARHL4VA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.171007360 (92h9m51.042090405s) [noise=true , address=81.171.14.70:4035 , piece_id=5T5MX4YDMI4IUY4LDSREN2MTN3NUL4GS6KZXDMNZMKJYY3VMS47Q, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.184999573 (92h9m51.056082648s) [noise=true , address=81.171.14.70:4032 , piece_id=GOAKND7Z3IDBUYT3ILUUBU7YC3UMA6PPDJLLMFTH2OKHF76GCHWA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.196460517 (92h9m51.067543562s) [noise=true , address=81.171.14.70:4040 , piece_id=4GA5CWKM5RJBL2IMWZNRGAA2T2JTFX54AFEFI3NAV2GNTKIDUELQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.199311115 (92h9m51.07039416s) [noise=true , address=81.171.14.70:4037 , piece_id=G6MBAEOMB7R4MQKZDULZY4DYQ5NR2QMCKW6HWYA55VD5YMSAERPA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.237519992 (92h9m51.108603028s) [noise=true , address=81.171.14.70:4030 , piece_id=VFAGZYZMOZKEHRNPUR2D2JCRDZFUFXUOS3JTTI2UOUYTIUY74NFQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.733107833 (92h9m51.604190868s) [noise=true , address=89.149.201.147:4005 , piece_id=OCHDE5VQIQPUVCOFXX343C7LXUIJFD4Q62ZZPEXKRN63KARUQR5A, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.815947736 (92h9m51.687030771s) [noise=true , address=89.149.201.147:4015 , piece_id=PGXJFJBHXVMXBTIWDTAOZKAU2JKTFSGCAD5I5MWU2MD32GF3AMDQ, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.841035780 (92h9m51.712118825s) [noise=true , address=89.149.201.147:4020 , piece_id=KIYFOE4WT3CGQQ3CL3BP2A3HH2KVKKLAUUGMHYPADVKI7BVBBRTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864720530 (92h9m51.735803575s) [noise=true , address=89.149.201.147:4002 , piece_id=TIK5YZA7N6MYGZ6KGNXPFBEQXD3DE2UXQ5Q2JP3EZGSLJ7TAO2GA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: <nil> Dec 9 11:38:51.864741660 (92h9m51.735824705s) [seg_pos={0 0}, upload=80368]: Segment reached optimal threshold of 26 pieces. Dec 9 11:38:51.865018136 (92h9m51.736101181s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PcxodnQMJERGjvBgnz3iEiC9MseRhMydkCriwGpQKqi6DGFDg): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865029489 (92h9m51.736112534s) [noise=true , address=89.149.201.147:4011 , piece_id=IJZKML6ZS5AF7KRYSHG4ZDR7AZSJGUWRSPTLSYTWR6WKYXWORAYA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865308640 (92h9m51.736391685s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 11:38:51.865331703 (92h9m51.736414748s) [noise=true , address=89.149.201.147:4007 , piece_id=FHFZMTRZEYLGK2QOXICD5VLT77MIFP3HL6Z7SP7URQEY5L3HIDTA, seg_pos={0 0}, upload=80368]: Long tail cancellation due to optimal threshold Dec 9 11:38:51.865403281 (92h9m51.736486326s) [seg_pos={0 0}, upload=80368]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 11:38:51.865634440 (92h9m51.736717485s) [upload=80368]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 11:38:51.865643307 (92h9m51.736726352s) [upload=80368]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 11:40:37.870707645 (92h11m37.741790681s) [upload=80368]: Done uploading segments... Dec 9 11:40:37.870754294 (92h11m37.741837339s) [upload=80368]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.159709846s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 1m0.000665021s trace: https://jaeger.dev.storj.io/trace/1947f0931d35a9e7 FAIL (2025-12-09 12:19:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00071694s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............XX................X.....X..........X.....X...] duration: 3.502562913s trace: https://jaeger.dev.storj.io/trace/508a3a20772980dc FAIL (2025-12-09 12:18:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.502591449s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [................XX................X.....X..........X.....X..] duration: 3.476244479s trace: https://jaeger.dev.storj.io/trace/508a3a20772980dc FAIL (2025-12-09 12:18:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.476280145s core uplink us-select-1 us1-placement-20

1
...
11612
11613
11614
11615
11616
...
17389