Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.671653247s trace: https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441 FAIL (2025-12-29 02:26:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.671682522s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001275745s trace: https://jaeger.dev.storj.io/trace/1c6bd0ca988ba441 FAIL (2025-12-29 02:25:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001297929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [........XX....X.....X......X...X....X.....X.........X.X.....] duration: 8.772950522s trace: https://jaeger.dev.storj.io/trace/75fe87cd46204a26 FAIL (2025-12-29 02:25:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA) context deadline exceeded ======LOGS====== Dec 29 02:21:01.634363389 (562h52m1.505446434s) [upload=493489]: Starting upload Dec 29 02:21:01.634468760 (562h52m1.505551815s) [upload=493489]: Done starting upload Dec 29 02:21:01.634556657 (562h52m1.505639702s) [upload=493489]: Uploading segments... Dec 29 02:21:01.635053473 (562h52m1.506136518s) [upload=493489]: Got next segment. Inline: false Dec 29 02:21:01.635115651 (562h52m1.506198696s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:21:01.635123407 (562h52m1.506206452s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:21:01.880859328 (562h52m1.751942363s) [seg_pos={0 0}, upload=493489]: Begin upload segment... Dec 29 02:21:01.880925895 (562h52m1.75200893s) [seg_pos={0 0}, upload=493489]: Done begin upload segment. Dec 29 02:21:01.880980970 (562h52m1.752064005s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881021908 (562h52m1.752104953s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881691493 (562h52m1.752774538s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882241640 (562h52m1.753324695s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882711915 (562h52m1.75379495s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883319954 (562h52m1.754402999s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883790449 (562h52m1.754873494s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.884342531 (562h52m1.755425576s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.885319922 (562h52m1.756402967s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.886011479 (562h52m1.757094524s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.887074052 (562h52m1.758157097s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.888011647 (562h52m1.759094692s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.889185552 (562h52m1.760268587s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.890297460 (562h52m1.761380515s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.891484380 (562h52m1.762567455s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.893535926 (562h52m1.764618981s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:03.613852650 (562h52m3.484935695s) [upload=493489]: Next returned nil segment Dec 29 02:21:03.613875022 (562h52m3.484958067s) [upload=493489]: Waiting for error group managing segments... Dec 29 02:21:03.894053117 (562h52m3.765136162s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.896711008 (562h52m3.767794043s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901245081 (562h52m3.772328126s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901930166 (562h52m3.773013211s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.907952763 (562h52m3.779035798s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913520245 (562h52m3.78460329s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913792213 (562h52m3.784875258s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.922567562 (562h52m3.793650607s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.923913344 (562h52m3.794996379s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.924247992 (562h52m3.795331037s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.926947782 (562h52m3.798030827s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.045580243 (562h52m3.916663288s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056933672 (562h52m3.928016717s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056958569 (562h52m3.928041615s) [seg_pos={0 0}, upload=493489]: Segment reached optimal threshold of 13 pieces. Dec 29 02:21:04.057191943 (562h52m3.928274988s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057201120 (562h52m3.928284165s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057213905 (562h52m3.92829696s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057226970 (562h52m3.928310015s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057454133 (562h52m3.928537178s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection Dec 29 02:21:04.057463601 (562h52m3.928546646s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057484300 (562h52m3.928567346s) [seg_pos={0 0}, upload=493489]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:21:04.057567358 (562h52m3.928650403s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:21:04.057575654 (562h52m3.928658699s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:21:04.348990433 (562h52m4.220073479s) [upload=493489]: Done uploading segments... Dec 29 02:21:04.349030249 (562h52m4.220113304s) [upload=493489]: Upload finished. err: <nil> Duration: 8.772972083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.........XX....X.....X......X...X....X.....X.........X.X....] duration: 6.008206695s trace: https://jaeger.dev.storj.io/trace/75fe87cd46204a26 FAIL (2025-12-29 02:24:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA) context deadline exceeded ======LOGS====== Dec 29 02:21:01.634363389 (562h52m1.505446434s) [upload=493489]: Starting upload Dec 29 02:21:01.634468760 (562h52m1.505551815s) [upload=493489]: Done starting upload Dec 29 02:21:01.634556657 (562h52m1.505639702s) [upload=493489]: Uploading segments... Dec 29 02:21:01.635053473 (562h52m1.506136518s) [upload=493489]: Got next segment. Inline: false Dec 29 02:21:01.635115651 (562h52m1.506198696s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:21:01.635123407 (562h52m1.506206452s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:21:01.880859328 (562h52m1.751942363s) [seg_pos={0 0}, upload=493489]: Begin upload segment... Dec 29 02:21:01.880925895 (562h52m1.75200893s) [seg_pos={0 0}, upload=493489]: Done begin upload segment. Dec 29 02:21:01.880980970 (562h52m1.752064005s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881021908 (562h52m1.752104953s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881691493 (562h52m1.752774538s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882241640 (562h52m1.753324695s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882711915 (562h52m1.75379495s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883319954 (562h52m1.754402999s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883790449 (562h52m1.754873494s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.884342531 (562h52m1.755425576s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.885319922 (562h52m1.756402967s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.886011479 (562h52m1.757094524s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.887074052 (562h52m1.758157097s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.888011647 (562h52m1.759094692s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.889185552 (562h52m1.760268587s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.890297460 (562h52m1.761380515s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.891484380 (562h52m1.762567455s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.893535926 (562h52m1.764618981s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:03.613852650 (562h52m3.484935695s) [upload=493489]: Next returned nil segment Dec 29 02:21:03.613875022 (562h52m3.484958067s) [upload=493489]: Waiting for error group managing segments... Dec 29 02:21:03.894053117 (562h52m3.765136162s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.896711008 (562h52m3.767794043s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901245081 (562h52m3.772328126s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901930166 (562h52m3.773013211s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.907952763 (562h52m3.779035798s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913520245 (562h52m3.78460329s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913792213 (562h52m3.784875258s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.922567562 (562h52m3.793650607s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.923913344 (562h52m3.794996379s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.924247992 (562h52m3.795331037s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.926947782 (562h52m3.798030827s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.045580243 (562h52m3.916663288s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056933672 (562h52m3.928016717s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056958569 (562h52m3.928041615s) [seg_pos={0 0}, upload=493489]: Segment reached optimal threshold of 13 pieces. Dec 29 02:21:04.057191943 (562h52m3.928274988s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057201120 (562h52m3.928284165s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057213905 (562h52m3.92829696s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057226970 (562h52m3.928310015s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057454133 (562h52m3.928537178s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection Dec 29 02:21:04.057463601 (562h52m3.928546646s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057484300 (562h52m3.928567346s) [seg_pos={0 0}, upload=493489]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:21:04.057567358 (562h52m3.928650403s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:21:04.057575654 (562h52m3.928658699s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:21:04.348990433 (562h52m4.220073479s) [upload=493489]: Done uploading segments... Dec 29 02:21:04.349030249 (562h52m4.220113304s) [upload=493489]: Upload finished. err: <nil> Duration: 6.008265186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..........XX....X.....X......X...X....X.....X.........X.X...] duration: 6.010457084s trace: https://jaeger.dev.storj.io/trace/75fe87cd46204a26 FAIL (2025-12-29 02:23:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA) context deadline exceeded ======LOGS====== Dec 29 02:21:01.634363389 (562h52m1.505446434s) [upload=493489]: Starting upload Dec 29 02:21:01.634468760 (562h52m1.505551815s) [upload=493489]: Done starting upload Dec 29 02:21:01.634556657 (562h52m1.505639702s) [upload=493489]: Uploading segments... Dec 29 02:21:01.635053473 (562h52m1.506136518s) [upload=493489]: Got next segment. Inline: false Dec 29 02:21:01.635115651 (562h52m1.506198696s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:21:01.635123407 (562h52m1.506206452s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:21:01.880859328 (562h52m1.751942363s) [seg_pos={0 0}, upload=493489]: Begin upload segment... Dec 29 02:21:01.880925895 (562h52m1.75200893s) [seg_pos={0 0}, upload=493489]: Done begin upload segment. Dec 29 02:21:01.880980970 (562h52m1.752064005s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881021908 (562h52m1.752104953s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881691493 (562h52m1.752774538s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882241640 (562h52m1.753324695s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882711915 (562h52m1.75379495s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883319954 (562h52m1.754402999s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883790449 (562h52m1.754873494s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.884342531 (562h52m1.755425576s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.885319922 (562h52m1.756402967s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.886011479 (562h52m1.757094524s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.887074052 (562h52m1.758157097s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.888011647 (562h52m1.759094692s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.889185552 (562h52m1.760268587s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.890297460 (562h52m1.761380515s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.891484380 (562h52m1.762567455s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.893535926 (562h52m1.764618981s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:03.613852650 (562h52m3.484935695s) [upload=493489]: Next returned nil segment Dec 29 02:21:03.613875022 (562h52m3.484958067s) [upload=493489]: Waiting for error group managing segments... Dec 29 02:21:03.894053117 (562h52m3.765136162s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.896711008 (562h52m3.767794043s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901245081 (562h52m3.772328126s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901930166 (562h52m3.773013211s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.907952763 (562h52m3.779035798s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913520245 (562h52m3.78460329s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913792213 (562h52m3.784875258s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.922567562 (562h52m3.793650607s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.923913344 (562h52m3.794996379s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.924247992 (562h52m3.795331037s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.926947782 (562h52m3.798030827s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.045580243 (562h52m3.916663288s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056933672 (562h52m3.928016717s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056958569 (562h52m3.928041615s) [seg_pos={0 0}, upload=493489]: Segment reached optimal threshold of 13 pieces. Dec 29 02:21:04.057191943 (562h52m3.928274988s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057201120 (562h52m3.928284165s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057213905 (562h52m3.92829696s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057226970 (562h52m3.928310015s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057454133 (562h52m3.928537178s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection Dec 29 02:21:04.057463601 (562h52m3.928546646s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057484300 (562h52m3.928567346s) [seg_pos={0 0}, upload=493489]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:21:04.057567358 (562h52m3.928650403s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:21:04.057575654 (562h52m3.928658699s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:21:04.348990433 (562h52m4.220073479s) [upload=493489]: Done uploading segments... Dec 29 02:21:04.349030249 (562h52m4.220113304s) [upload=493489]: Upload finished. err: <nil> Duration: 6.010486762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...........XX....X.....X......X...X....X.....X.........X.X..] duration: 5.91191695s trace: https://jaeger.dev.storj.io/trace/75fe87cd46204a26 FAIL (2025-12-29 02:22:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA) context deadline exceeded ======LOGS====== Dec 29 02:21:01.634363389 (562h52m1.505446434s) [upload=493489]: Starting upload Dec 29 02:21:01.634468760 (562h52m1.505551815s) [upload=493489]: Done starting upload Dec 29 02:21:01.634556657 (562h52m1.505639702s) [upload=493489]: Uploading segments... Dec 29 02:21:01.635053473 (562h52m1.506136518s) [upload=493489]: Got next segment. Inline: false Dec 29 02:21:01.635115651 (562h52m1.506198696s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:21:01.635123407 (562h52m1.506206452s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:21:01.880859328 (562h52m1.751942363s) [seg_pos={0 0}, upload=493489]: Begin upload segment... Dec 29 02:21:01.880925895 (562h52m1.75200893s) [seg_pos={0 0}, upload=493489]: Done begin upload segment. Dec 29 02:21:01.880980970 (562h52m1.752064005s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881021908 (562h52m1.752104953s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881691493 (562h52m1.752774538s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882241640 (562h52m1.753324695s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882711915 (562h52m1.75379495s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883319954 (562h52m1.754402999s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883790449 (562h52m1.754873494s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.884342531 (562h52m1.755425576s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.885319922 (562h52m1.756402967s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.886011479 (562h52m1.757094524s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.887074052 (562h52m1.758157097s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.888011647 (562h52m1.759094692s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.889185552 (562h52m1.760268587s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.890297460 (562h52m1.761380515s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.891484380 (562h52m1.762567455s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.893535926 (562h52m1.764618981s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:03.613852650 (562h52m3.484935695s) [upload=493489]: Next returned nil segment Dec 29 02:21:03.613875022 (562h52m3.484958067s) [upload=493489]: Waiting for error group managing segments... Dec 29 02:21:03.894053117 (562h52m3.765136162s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.896711008 (562h52m3.767794043s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901245081 (562h52m3.772328126s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901930166 (562h52m3.773013211s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.907952763 (562h52m3.779035798s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913520245 (562h52m3.78460329s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913792213 (562h52m3.784875258s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.922567562 (562h52m3.793650607s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.923913344 (562h52m3.794996379s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.924247992 (562h52m3.795331037s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.926947782 (562h52m3.798030827s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.045580243 (562h52m3.916663288s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056933672 (562h52m3.928016717s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056958569 (562h52m3.928041615s) [seg_pos={0 0}, upload=493489]: Segment reached optimal threshold of 13 pieces. Dec 29 02:21:04.057191943 (562h52m3.928274988s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057201120 (562h52m3.928284165s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057213905 (562h52m3.92829696s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057226970 (562h52m3.928310015s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057454133 (562h52m3.928537178s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection Dec 29 02:21:04.057463601 (562h52m3.928546646s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057484300 (562h52m3.928567346s) [seg_pos={0 0}, upload=493489]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:21:04.057567358 (562h52m3.928650403s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:21:04.057575654 (562h52m3.928658699s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:21:04.348990433 (562h52m4.220073479s) [upload=493489]: Done uploading segments... Dec 29 02:21:04.349030249 (562h52m4.220113304s) [upload=493489]: Upload finished. err: <nil> Duration: 5.911946495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [............XX....X.....X......X...X....X.....X.........X.X.] duration: 11.645261989s trace: https://jaeger.dev.storj.io/trace/75fe87cd46204a26 FAIL (2025-12-29 02:21:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA) context deadline exceeded ======LOGS====== Dec 29 02:21:01.634363389 (562h52m1.505446434s) [upload=493489]: Starting upload Dec 29 02:21:01.634468760 (562h52m1.505551815s) [upload=493489]: Done starting upload Dec 29 02:21:01.634556657 (562h52m1.505639702s) [upload=493489]: Uploading segments... Dec 29 02:21:01.635053473 (562h52m1.506136518s) [upload=493489]: Got next segment. Inline: false Dec 29 02:21:01.635115651 (562h52m1.506198696s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:21:01.635123407 (562h52m1.506206452s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:21:01.880859328 (562h52m1.751942363s) [seg_pos={0 0}, upload=493489]: Begin upload segment... Dec 29 02:21:01.880925895 (562h52m1.75200893s) [seg_pos={0 0}, upload=493489]: Done begin upload segment. Dec 29 02:21:01.880980970 (562h52m1.752064005s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881021908 (562h52m1.752104953s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881691493 (562h52m1.752774538s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882241640 (562h52m1.753324695s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882711915 (562h52m1.75379495s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883319954 (562h52m1.754402999s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883790449 (562h52m1.754873494s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.884342531 (562h52m1.755425576s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.885319922 (562h52m1.756402967s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.886011479 (562h52m1.757094524s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.887074052 (562h52m1.758157097s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.888011647 (562h52m1.759094692s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.889185552 (562h52m1.760268587s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.890297460 (562h52m1.761380515s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.891484380 (562h52m1.762567455s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.893535926 (562h52m1.764618981s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:03.613852650 (562h52m3.484935695s) [upload=493489]: Next returned nil segment Dec 29 02:21:03.613875022 (562h52m3.484958067s) [upload=493489]: Waiting for error group managing segments... Dec 29 02:21:03.894053117 (562h52m3.765136162s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.896711008 (562h52m3.767794043s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901245081 (562h52m3.772328126s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901930166 (562h52m3.773013211s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.907952763 (562h52m3.779035798s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913520245 (562h52m3.78460329s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913792213 (562h52m3.784875258s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.922567562 (562h52m3.793650607s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.923913344 (562h52m3.794996379s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.924247992 (562h52m3.795331037s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.926947782 (562h52m3.798030827s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.045580243 (562h52m3.916663288s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056933672 (562h52m3.928016717s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056958569 (562h52m3.928041615s) [seg_pos={0 0}, upload=493489]: Segment reached optimal threshold of 13 pieces. Dec 29 02:21:04.057191943 (562h52m3.928274988s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057201120 (562h52m3.928284165s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057213905 (562h52m3.92829696s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057226970 (562h52m3.928310015s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057454133 (562h52m3.928537178s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection Dec 29 02:21:04.057463601 (562h52m3.928546646s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057484300 (562h52m3.928567346s) [seg_pos={0 0}, upload=493489]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:21:04.057567358 (562h52m3.928650403s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:21:04.057575654 (562h52m3.928658699s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:21:04.348990433 (562h52m4.220073479s) [upload=493489]: Done uploading segments... Dec 29 02:21:04.349030249 (562h52m4.220113304s) [upload=493489]: Upload finished. err: <nil> Duration: 11.64533631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.............XX....X.....X......X...X....X.....X.........X.X] duration: 30.003128248s trace: https://jaeger.dev.storj.io/trace/75fe87cd46204a26 FAIL (2025-12-29 02:21:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA) context deadline exceeded ======LOGS====== Dec 29 02:21:01.634363389 (562h52m1.505446434s) [upload=493489]: Starting upload Dec 29 02:21:01.634468760 (562h52m1.505551815s) [upload=493489]: Done starting upload Dec 29 02:21:01.634556657 (562h52m1.505639702s) [upload=493489]: Uploading segments... Dec 29 02:21:01.635053473 (562h52m1.506136518s) [upload=493489]: Got next segment. Inline: false Dec 29 02:21:01.635115651 (562h52m1.506198696s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:21:01.635123407 (562h52m1.506206452s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:21:01.880859328 (562h52m1.751942363s) [seg_pos={0 0}, upload=493489]: Begin upload segment... Dec 29 02:21:01.880925895 (562h52m1.75200893s) [seg_pos={0 0}, upload=493489]: Done begin upload segment. Dec 29 02:21:01.880980970 (562h52m1.752064005s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881021908 (562h52m1.752104953s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.881691493 (562h52m1.752774538s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882241640 (562h52m1.753324695s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.882711915 (562h52m1.75379495s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883319954 (562h52m1.754402999s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.883790449 (562h52m1.754873494s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.884342531 (562h52m1.755425576s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.885319922 (562h52m1.756402967s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.886011479 (562h52m1.757094524s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.887074052 (562h52m1.758157097s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.888011647 (562h52m1.759094692s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.889185552 (562h52m1.760268587s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.890297460 (562h52m1.761380515s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.891484380 (562h52m1.762567455s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:01.893535926 (562h52m1.764618981s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Uploading piece... Dec 29 02:21:03.613852650 (562h52m3.484935695s) [upload=493489]: Next returned nil segment Dec 29 02:21:03.613875022 (562h52m3.484958067s) [upload=493489]: Waiting for error group managing segments... Dec 29 02:21:03.894053117 (562h52m3.765136162s) [noise=true , address=145.79.227.12:4767 , piece_id=K3UE6QQ5TSCLH5TXTWFSZSVFFDPM6B2HKL5MDMLBGLZYHRB3776A, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.896711008 (562h52m3.767794043s) [noise=true , address=145.79.227.11:4729 , piece_id=QRPYR7QWJXFLJREXZVSN2SHTQ7EZXXEWNEUXLLXNQ7LQ4WWPUIYQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901245081 (562h52m3.772328126s) [noise=true , address=145.79.227.7:4293 , piece_id=WK2YYVQTMPFVKYFBJQLM5JZG35GBDKNYF73VLCAFSXEJDYRRVAUA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.901930166 (562h52m3.773013211s) [noise=true , address=145.79.227.21:4473 , piece_id=Q5BWDN2AEJ2YU5T7ILHHXIKBC3SSKQPHQXYSWPZLKZOJVW4D65VQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.907952763 (562h52m3.779035798s) [noise=true , address=145.79.227.48:4301 , piece_id=J3DFQF7DECDWUQ4PQJEJMANKXUXQYKQKTICS6TMZ2HBTP4WQBVJQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913520245 (562h52m3.78460329s) [noise=true , address=145.79.227.3:4151 , piece_id=H7VTKBR4PYWVOOGXXHQXTIR7MOGZHDFX5A5S7G7N2QAFCNCWSLSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.913792213 (562h52m3.784875258s) [noise=true , address=145.79.227.39:4014 , piece_id=77KJBP7HNPV3YGCFQ3LTNBNEO5DIGINZS3JUQNDUXMIY4EWUYOXQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.922567562 (562h52m3.793650607s) [noise=true , address=145.79.227.50:4375 , piece_id=H34NLA7HFFURD4JKSVTFJYQQE35B4D22XWULE7UXXUDED2EGSU2Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.923913344 (562h52m3.794996379s) [noise=true , address=145.79.227.51:4411 , piece_id=4SO4HA7OQ6XXJ2Z37V4JOKPV2W5GCOGGFC5ZUZMRZETFYCTWZYLA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.924247992 (562h52m3.795331037s) [noise=true , address=145.79.227.42:4122 , piece_id=ALFKPZXS32SOJHFW3ZLSGNF6FJOCRGXCWGU5AZC5AUTICNICJLCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:03.926947782 (562h52m3.798030827s) [noise=true , address=145.79.227.45:4229 , piece_id=A5LGNUPYGTUVTEFCLWDHYFSLTW5NR36UP3Q7DFDXJ5QVRVV3NF5Q, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.045580243 (562h52m3.916663288s) [noise=true , address=77.90.31.13:4627 , piece_id=EDMOPSFOHHQM6K5U3ZU7XPMQH3UTPWJHY5OET4RW6BBYPUFKAKQA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056933672 (562h52m3.928016717s) [noise=true , address=77.90.31.18:4805 , piece_id=FMB37EDSFKM5JFJVPALHTGZVNPQIQMW4H4PNL2A7QIQ2JGZT2EPQ, seg_pos={0 0}, upload=493489]: Done uploading piece. err: <nil> Dec 29 02:21:04.056958569 (562h52m3.928041615s) [seg_pos={0 0}, upload=493489]: Segment reached optimal threshold of 13 pieces. Dec 29 02:21:04.057191943 (562h52m3.928274988s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057201120 (562h52m3.928284165s) [noise=true , address=45.91.116.9:4264 , piece_id=W4LW576LRT7AOQ2JDBXEHRDGZP2MZ5KUSM3BIG45B47FUTFXUJSA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057213905 (562h52m3.92829696s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 02:21:04.057226970 (562h52m3.928310015s) [noise=true , address=45.91.116.12:4228 , piece_id=5BAHSBRCJJRJDGMMDG4KQ3Y7MCH54MQWF2JQMOLOUY7AQP3ARCCA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057454133 (562h52m3.928537178s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42146->45.91.116.4:4937: use of closed network connection Dec 29 02:21:04.057463601 (562h52m3.928546646s) [noise=true , address=45.91.116.4:4937 , piece_id=TTTH36LLNUIVJIGVWYZQQ2PFOCJHWKLRLHDJVFZ4XLB6FA4WAPBA, seg_pos={0 0}, upload=493489]: Long tail cancellation due to optimal threshold Dec 29 02:21:04.057484300 (562h52m3.928567346s) [seg_pos={0 0}, upload=493489]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:21:04.057567358 (562h52m3.928650403s) [upload=493489]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:21:04.057575654 (562h52m3.928658699s) [upload=493489]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:21:04.348990433 (562h52m4.220073479s) [upload=493489]: Done uploading segments... Dec 29 02:21:04.349030249 (562h52m4.220113304s) [upload=493489]: Upload finished. err: <nil> Duration: 30.003150442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..............XX....X.....X......X...X....X.....X.........X.] duration: 9.010645251s trace: https://jaeger.dev.storj.io/trace/21d6fef344aacb41 FAIL (2025-12-29 02:20:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: S5KDXA6RLPLTUHAKFAD4ZCG4NJJ657EPGRBQX7I3EBXPOZOOCCKQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: OKG6TQUZN5ZSKL6GDCJ2MSPBE5RWJAASE2CV47CBNSTPJX5SH6TA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: AZ7HE646VG4SMCN5VXJHS3T3WR3SEC3XU5IZOPX6HQWIFCJ7IXJQ) context deadline exceeded ======LOGS====== Dec 29 02:19:17.325572487 (562h50m17.196655522s) [upload=493469]: Starting upload Dec 29 02:19:17.325632981 (562h50m17.196716026s) [upload=493469]: Done starting upload Dec 29 02:19:17.325853041 (562h50m17.196936086s) [upload=493469]: Uploading segments... Dec 29 02:19:17.325920899 (562h50m17.197003944s) [upload=493469]: Got next segment. Inline: false Dec 29 02:19:17.325959794 (562h50m17.197042839s) [upload=493469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:19:17.325966236 (562h50m17.197049281s) [upload=493469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:19:17.577491961 (562h50m17.448575006s) [seg_pos={0 0}, upload=493469]: Begin upload segment... Dec 29 02:19:17.577550834 (562h50m17.448633869s) [seg_pos={0 0}, upload=493469]: Done begin upload segment. Dec 29 02:19:17.577612131 (562h50m17.448695166s) [noise=true , address=145.79.227.52:4445 , piece_id=3CYOWAMENK75DXSF2BMSF6HIYJF344TYVKU5YJX7VU47RXJG6BCA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.577647818 (562h50m17.448730854s) [noise=true , address=77.90.31.18:4805 , piece_id=AZ7HE646VG4SMCN5VXJHS3T3WR3SEC3XU5IZOPX6HQWIFCJ7IXJQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.578199179 (562h50m17.449282214s) [noise=true , address=77.90.31.11:4556 , piece_id=BWKYLBG5B647AKDE7KVWKZYL4GDKRAKKBBXI7NHNN35TKNUCR5BA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.578600673 (562h50m17.449683708s) [noise=true , address=45.91.116.5:4300 , piece_id=VCHBIHOFYECFDVGQU3VP4UFTZ2FYQ2RAI3A7HLELMMSIIFBRKOFQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.579055939 (562h50m17.450138974s) [noise=true , address=77.90.31.21:4880 , piece_id=QA3E7LM373UV4OZN4JJW4OCVBYM33GC3F2I3XXGFSCKQRGJM6EVQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.579409793 (562h50m17.450492828s) [noise=true , address=145.79.227.36:4518 , piece_id=HDCTSLC74GZJBKRSNDNZ2FHUUJRO2WI43OGSROUP2IEZM7A5XXWA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.579731506 (562h50m17.450814551s) [noise=true , address=145.79.227.51:4410 , piece_id=325UFUNVVV3UQRYEC5WH2QCCDLI6CVXCWHCGQTEHHNGPFDUDF3BQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.580005387 (562h50m17.451088422s) [noise=true , address=77.90.31.12:4590 , piece_id=BPDGNSD7Q3UCRK3JHVT6JVILU4DFZM7VNB72MTVNXMZNGY66T7JA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.580303425 (562h50m17.45138646s) [noise=true , address=145.79.227.35:4481 , piece_id=MILJWD4WECMT6CS7MH4FEUOKF6IIE34SMI3HZPDANRP7KBBCVOBA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.580623815 (562h50m17.45170685s) [noise=true , address=145.79.227.43:4159 , piece_id=ZLC27E4JZCW2I5GW6YMDCNEAI4HA7GD5RWZNIG6UF35CG7FGZQIQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.580918637 (562h50m17.452001672s) [noise=true , address=145.79.227.13:4802 , piece_id=OU5673OTOGNVQ65RZP4OC74QXC5YBSE62BGZ66D3EQ7VOBLBPGWQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.581228537 (562h50m17.452311572s) [noise=true , address=145.79.227.33:4870 , piece_id=YOTK5TNY2LKLXKIGDHO5CLUEDY4VLV5L3ANM2TS2Y6GWMPHHUEBQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.582030123 (562h50m17.453113168s) [noise=true , address=45.91.116.7:4011 , piece_id=S5KDXA6RLPLTUHAKFAD4ZCG4NJJ657EPGRBQX7I3EBXPOZOOCCKQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.584037585 (562h50m17.45512063s) [noise=true , address=77.90.31.17:4770 , piece_id=XYP4P2JURCTIAVBTJ5TROSLDXBMOZ6EN64FEW2T2YN3JUAGSVYEA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.584547116 (562h50m17.455630191s) [noise=true , address=77.90.31.15:4700 , piece_id=OKG6TQUZN5ZSKL6GDCJ2MSPBE5RWJAASE2CV47CBNSTPJX5SH6TA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.584993476 (562h50m17.456076511s) [noise=true , address=145.79.227.29:4943 , piece_id=NWDFY5ZDX3FLJ6GMCQTILCF7BHTASDV443KLWAWGTHF34Q7XB4KA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:19.295935142 (562h50m19.167018208s) [upload=493469]: Next returned nil segment Dec 29 02:19:19.295950953 (562h50m19.167033988s) [upload=493469]: Waiting for error group managing segments... Dec 29 02:19:19.586511787 (562h50m19.457594832s) [noise=true , address=145.79.227.36:4518 , piece_id=HDCTSLC74GZJBKRSNDNZ2FHUUJRO2WI43OGSROUP2IEZM7A5XXWA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.586991491 (562h50m19.458074536s) [noise=true , address=145.79.227.52:4445 , piece_id=3CYOWAMENK75DXSF2BMSF6HIYJF344TYVKU5YJX7VU47RXJG6BCA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.589111206 (562h50m19.460194241s) [noise=true , address=145.79.227.13:4802 , piece_id=OU5673OTOGNVQ65RZP4OC74QXC5YBSE62BGZ66D3EQ7VOBLBPGWQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.589840164 (562h50m19.4609232s) [noise=true , address=145.79.227.51:4410 , piece_id=325UFUNVVV3UQRYEC5WH2QCCDLI6CVXCWHCGQTEHHNGPFDUDF3BQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.745512214 (562h50m19.616595249s) [noise=true , address=77.90.31.18:4805 , piece_id=AZ7HE646VG4SMCN5VXJHS3T3WR3SEC3XU5IZOPX6HQWIFCJ7IXJQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.805144030 (562h50m19.676227095s) [noise=true , address=77.90.31.17:4770 , piece_id=XYP4P2JURCTIAVBTJ5TROSLDXBMOZ6EN64FEW2T2YN3JUAGSVYEA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.836367423 (562h50m19.707450468s) [noise=true , address=77.90.31.11:4556 , piece_id=BWKYLBG5B647AKDE7KVWKZYL4GDKRAKKBBXI7NHNN35TKNUCR5BA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.909185878 (562h50m19.780268923s) [noise=true , address=45.91.116.5:4300 , piece_id=VCHBIHOFYECFDVGQU3VP4UFTZ2FYQ2RAI3A7HLELMMSIIFBRKOFQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.932205620 (562h50m19.803288655s) [noise=true , address=45.91.116.7:4011 , piece_id=S5KDXA6RLPLTUHAKFAD4ZCG4NJJ657EPGRBQX7I3EBXPOZOOCCKQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:20.425156580 (562h50m20.296239625s) [noise=true , address=77.90.31.15:4700 , piece_id=OKG6TQUZN5ZSKL6GDCJ2MSPBE5RWJAASE2CV47CBNSTPJX5SH6TA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:20.443928695 (562h50m20.31501174s) [noise=true , address=77.90.31.12:4590 , piece_id=BPDGNSD7Q3UCRK3JHVT6JVILU4DFZM7VNB72MTVNXMZNGY66T7JA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:20.693271606 (562h50m20.564354641s) [noise=true , address=77.90.31.21:4880 , piece_id=QA3E7LM373UV4OZN4JJW4OCVBYM33GC3F2I3XXGFSCKQRGJM6EVQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:20.898760721 (562h50m20.769843766s) [noise=true , address=145.79.227.43:4159 , piece_id=ZLC27E4JZCW2I5GW6YMDCNEAI4HA7GD5RWZNIG6UF35CG7FGZQIQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:20.898785849 (562h50m20.769868884s) [seg_pos={0 0}, upload=493469]: Segment reached optimal threshold of 13 pieces. Dec 29 02:19:20.899141587 (562h50m20.770224622s) [noise=true , address=145.79.227.29:4943 , piece_id=NWDFY5ZDX3FLJ6GMCQTILCF7BHTASDV443KLWAWGTHF34Q7XB4KA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 02:19:20.899149942 (562h50m20.770232987s) [noise=true , address=145.79.227.29:4943 , piece_id=NWDFY5ZDX3FLJ6GMCQTILCF7BHTASDV443KLWAWGTHF34Q7XB4KA, seg_pos={0 0}, upload=493469]: Long tail cancellation due to optimal threshold Dec 29 02:19:20.899197022 (562h50m20.770280067s) [noise=true , address=145.79.227.35:4481 , piece_id=MILJWD4WECMT6CS7MH4FEUOKF6IIE34SMI3HZPDANRP7KBBCVOBA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 02:19:20.899207312 (562h50m20.770290357s) [noise=true , address=145.79.227.35:4481 , piece_id=MILJWD4WECMT6CS7MH4FEUOKF6IIE34SMI3HZPDANRP7KBBCVOBA, seg_pos={0 0}, upload=493469]: Long tail cancellation due to optimal threshold Dec 29 02:19:20.899304526 (562h50m20.770387571s) [noise=true , address=145.79.227.33:4870 , piece_id=YOTK5TNY2LKLXKIGDHO5CLUEDY4VLV5L3ANM2TS2Y6GWMPHHUEBQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 29 02:19:20.899313223 (562h50m20.770396269s) [noise=true , address=145.79.227.33:4870 , piece_id=YOTK5TNY2LKLXKIGDHO5CLUEDY4VLV5L3ANM2TS2Y6GWMPHHUEBQ, seg_pos={0 0}, upload=493469]: Long tail cancellation due to optimal threshold Dec 29 02:19:20.899331538 (562h50m20.770414573s) [seg_pos={0 0}, upload=493469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:19:20.899408394 (562h50m20.770491429s) [upload=493469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:19:20.899414206 (562h50m20.770497251s) [upload=493469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:19:21.218564889 (562h50m21.089647934s) [upload=493469]: Done uploading segments... Dec 29 02:19:21.218584917 (562h50m21.089667952s) [upload=493469]: Upload finished. err: <nil> Duration: 9.010672592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...............XX....X.....X......X...X....X.....X.........X] duration: 30.002322954s trace: https://jaeger.dev.storj.io/trace/21d6fef344aacb41 FAIL (2025-12-29 02:19:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: S5KDXA6RLPLTUHAKFAD4ZCG4NJJ657EPGRBQX7I3EBXPOZOOCCKQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: OKG6TQUZN5ZSKL6GDCJ2MSPBE5RWJAASE2CV47CBNSTPJX5SH6TA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: AZ7HE646VG4SMCN5VXJHS3T3WR3SEC3XU5IZOPX6HQWIFCJ7IXJQ) context deadline exceeded ======LOGS====== Dec 29 02:19:17.325572487 (562h50m17.196655522s) [upload=493469]: Starting upload Dec 29 02:19:17.325632981 (562h50m17.196716026s) [upload=493469]: Done starting upload Dec 29 02:19:17.325853041 (562h50m17.196936086s) [upload=493469]: Uploading segments... Dec 29 02:19:17.325920899 (562h50m17.197003944s) [upload=493469]: Got next segment. Inline: false Dec 29 02:19:17.325959794 (562h50m17.197042839s) [upload=493469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:19:17.325966236 (562h50m17.197049281s) [upload=493469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:19:17.577491961 (562h50m17.448575006s) [seg_pos={0 0}, upload=493469]: Begin upload segment... Dec 29 02:19:17.577550834 (562h50m17.448633869s) [seg_pos={0 0}, upload=493469]: Done begin upload segment. Dec 29 02:19:17.577612131 (562h50m17.448695166s) [noise=true , address=145.79.227.52:4445 , piece_id=3CYOWAMENK75DXSF2BMSF6HIYJF344TYVKU5YJX7VU47RXJG6BCA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.577647818 (562h50m17.448730854s) [noise=true , address=77.90.31.18:4805 , piece_id=AZ7HE646VG4SMCN5VXJHS3T3WR3SEC3XU5IZOPX6HQWIFCJ7IXJQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.578199179 (562h50m17.449282214s) [noise=true , address=77.90.31.11:4556 , piece_id=BWKYLBG5B647AKDE7KVWKZYL4GDKRAKKBBXI7NHNN35TKNUCR5BA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.578600673 (562h50m17.449683708s) [noise=true , address=45.91.116.5:4300 , piece_id=VCHBIHOFYECFDVGQU3VP4UFTZ2FYQ2RAI3A7HLELMMSIIFBRKOFQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.579055939 (562h50m17.450138974s) [noise=true , address=77.90.31.21:4880 , piece_id=QA3E7LM373UV4OZN4JJW4OCVBYM33GC3F2I3XXGFSCKQRGJM6EVQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.579409793 (562h50m17.450492828s) [noise=true , address=145.79.227.36:4518 , piece_id=HDCTSLC74GZJBKRSNDNZ2FHUUJRO2WI43OGSROUP2IEZM7A5XXWA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.579731506 (562h50m17.450814551s) [noise=true , address=145.79.227.51:4410 , piece_id=325UFUNVVV3UQRYEC5WH2QCCDLI6CVXCWHCGQTEHHNGPFDUDF3BQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.580005387 (562h50m17.451088422s) [noise=true , address=77.90.31.12:4590 , piece_id=BPDGNSD7Q3UCRK3JHVT6JVILU4DFZM7VNB72MTVNXMZNGY66T7JA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.580303425 (562h50m17.45138646s) [noise=true , address=145.79.227.35:4481 , piece_id=MILJWD4WECMT6CS7MH4FEUOKF6IIE34SMI3HZPDANRP7KBBCVOBA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.580623815 (562h50m17.45170685s) [noise=true , address=145.79.227.43:4159 , piece_id=ZLC27E4JZCW2I5GW6YMDCNEAI4HA7GD5RWZNIG6UF35CG7FGZQIQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.580918637 (562h50m17.452001672s) [noise=true , address=145.79.227.13:4802 , piece_id=OU5673OTOGNVQ65RZP4OC74QXC5YBSE62BGZ66D3EQ7VOBLBPGWQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.581228537 (562h50m17.452311572s) [noise=true , address=145.79.227.33:4870 , piece_id=YOTK5TNY2LKLXKIGDHO5CLUEDY4VLV5L3ANM2TS2Y6GWMPHHUEBQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.582030123 (562h50m17.453113168s) [noise=true , address=45.91.116.7:4011 , piece_id=S5KDXA6RLPLTUHAKFAD4ZCG4NJJ657EPGRBQX7I3EBXPOZOOCCKQ, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.584037585 (562h50m17.45512063s) [noise=true , address=77.90.31.17:4770 , piece_id=XYP4P2JURCTIAVBTJ5TROSLDXBMOZ6EN64FEW2T2YN3JUAGSVYEA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.584547116 (562h50m17.455630191s) [noise=true , address=77.90.31.15:4700 , piece_id=OKG6TQUZN5ZSKL6GDCJ2MSPBE5RWJAASE2CV47CBNSTPJX5SH6TA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:17.584993476 (562h50m17.456076511s) [noise=true , address=145.79.227.29:4943 , piece_id=NWDFY5ZDX3FLJ6GMCQTILCF7BHTASDV443KLWAWGTHF34Q7XB4KA, seg_pos={0 0}, upload=493469]: Uploading piece... Dec 29 02:19:19.295935142 (562h50m19.167018208s) [upload=493469]: Next returned nil segment Dec 29 02:19:19.295950953 (562h50m19.167033988s) [upload=493469]: Waiting for error group managing segments... Dec 29 02:19:19.586511787 (562h50m19.457594832s) [noise=true , address=145.79.227.36:4518 , piece_id=HDCTSLC74GZJBKRSNDNZ2FHUUJRO2WI43OGSROUP2IEZM7A5XXWA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.586991491 (562h50m19.458074536s) [noise=true , address=145.79.227.52:4445 , piece_id=3CYOWAMENK75DXSF2BMSF6HIYJF344TYVKU5YJX7VU47RXJG6BCA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.589111206 (562h50m19.460194241s) [noise=true , address=145.79.227.13:4802 , piece_id=OU5673OTOGNVQ65RZP4OC74QXC5YBSE62BGZ66D3EQ7VOBLBPGWQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.589840164 (562h50m19.4609232s) [noise=true , address=145.79.227.51:4410 , piece_id=325UFUNVVV3UQRYEC5WH2QCCDLI6CVXCWHCGQTEHHNGPFDUDF3BQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.745512214 (562h50m19.616595249s) [noise=true , address=77.90.31.18:4805 , piece_id=AZ7HE646VG4SMCN5VXJHS3T3WR3SEC3XU5IZOPX6HQWIFCJ7IXJQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.805144030 (562h50m19.676227095s) [noise=true , address=77.90.31.17:4770 , piece_id=XYP4P2JURCTIAVBTJ5TROSLDXBMOZ6EN64FEW2T2YN3JUAGSVYEA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.836367423 (562h50m19.707450468s) [noise=true , address=77.90.31.11:4556 , piece_id=BWKYLBG5B647AKDE7KVWKZYL4GDKRAKKBBXI7NHNN35TKNUCR5BA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.909185878 (562h50m19.780268923s) [noise=true , address=45.91.116.5:4300 , piece_id=VCHBIHOFYECFDVGQU3VP4UFTZ2FYQ2RAI3A7HLELMMSIIFBRKOFQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:19.932205620 (562h50m19.803288655s) [noise=true , address=45.91.116.7:4011 , piece_id=S5KDXA6RLPLTUHAKFAD4ZCG4NJJ657EPGRBQX7I3EBXPOZOOCCKQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:20.425156580 (562h50m20.296239625s) [noise=true , address=77.90.31.15:4700 , piece_id=OKG6TQUZN5ZSKL6GDCJ2MSPBE5RWJAASE2CV47CBNSTPJX5SH6TA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:20.443928695 (562h50m20.31501174s) [noise=true , address=77.90.31.12:4590 , piece_id=BPDGNSD7Q3UCRK3JHVT6JVILU4DFZM7VNB72MTVNXMZNGY66T7JA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:20.693271606 (562h50m20.564354641s) [noise=true , address=77.90.31.21:4880 , piece_id=QA3E7LM373UV4OZN4JJW4OCVBYM33GC3F2I3XXGFSCKQRGJM6EVQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:20.898760721 (562h50m20.769843766s) [noise=true , address=145.79.227.43:4159 , piece_id=ZLC27E4JZCW2I5GW6YMDCNEAI4HA7GD5RWZNIG6UF35CG7FGZQIQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: <nil> Dec 29 02:19:20.898785849 (562h50m20.769868884s) [seg_pos={0 0}, upload=493469]: Segment reached optimal threshold of 13 pieces. Dec 29 02:19:20.899141587 (562h50m20.770224622s) [noise=true , address=145.79.227.29:4943 , piece_id=NWDFY5ZDX3FLJ6GMCQTILCF7BHTASDV443KLWAWGTHF34Q7XB4KA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 29 02:19:20.899149942 (562h50m20.770232987s) [noise=true , address=145.79.227.29:4943 , piece_id=NWDFY5ZDX3FLJ6GMCQTILCF7BHTASDV443KLWAWGTHF34Q7XB4KA, seg_pos={0 0}, upload=493469]: Long tail cancellation due to optimal threshold Dec 29 02:19:20.899197022 (562h50m20.770280067s) [noise=true , address=145.79.227.35:4481 , piece_id=MILJWD4WECMT6CS7MH4FEUOKF6IIE34SMI3HZPDANRP7KBBCVOBA, seg_pos={0 0}, upload=493469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 02:19:20.899207312 (562h50m20.770290357s) [noise=true , address=145.79.227.35:4481 , piece_id=MILJWD4WECMT6CS7MH4FEUOKF6IIE34SMI3HZPDANRP7KBBCVOBA, seg_pos={0 0}, upload=493469]: Long tail cancellation due to optimal threshold Dec 29 02:19:20.899304526 (562h50m20.770387571s) [noise=true , address=145.79.227.33:4870 , piece_id=YOTK5TNY2LKLXKIGDHO5CLUEDY4VLV5L3ANM2TS2Y6GWMPHHUEBQ, seg_pos={0 0}, upload=493469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 29 02:19:20.899313223 (562h50m20.770396269s) [noise=true , address=145.79.227.33:4870 , piece_id=YOTK5TNY2LKLXKIGDHO5CLUEDY4VLV5L3ANM2TS2Y6GWMPHHUEBQ, seg_pos={0 0}, upload=493469]: Long tail cancellation due to optimal threshold Dec 29 02:19:20.899331538 (562h50m20.770414573s) [seg_pos={0 0}, upload=493469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:19:20.899408394 (562h50m20.770491429s) [upload=493469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:19:20.899414206 (562h50m20.770497251s) [upload=493469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:19:21.218564889 (562h50m21.089647934s) [upload=493469]: Done uploading segments... Dec 29 02:19:21.218584917 (562h50m21.089667952s) [upload=493469]: Upload finished. err: <nil> Duration: 30.002345357s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................XX....X.....X......X...X....X.....X.........] duration: 6.54738188s trace: https://jaeger.dev.storj.io/trace/6457a1a4845279ab FAIL (2025-12-29 02:18:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (562h40m57.020745103s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:09:57.402920664 (562h40m57.274003699s) [seg_pos={0 0}, upload=493334]: Begin upload segment... Dec 29 02:09:57.402971150 (562h40m57.274054185s) [seg_pos={0 0}, upload=493334]: Done begin upload segment. Dec 29 02:09:57.403010204 (562h40m57.274093239s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403021486 (562h40m57.274104531s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403158967 (562h40m57.274242012s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403570060 (562h40m57.274653095s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403681472 (562h40m57.274764507s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403859191 (562h40m57.274942227s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404011882 (562h40m57.275094917s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404109588 (562h40m57.275192623s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404360806 (562h40m57.275443841s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404605462 (562h40m57.275688497s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404840620 (562h40m57.275923655s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405103179 (562h40m57.276186215s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405201777 (562h40m57.276284812s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405343958 (562h40m57.276426993s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405587281 (562h40m57.276670316s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405920556 (562h40m57.277003602s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:59.145565768 (562h40m59.016648803s) [upload=493334]: Next returned nil segment Dec 29 02:09:59.145581808 (562h40m59.016664853s) [upload=493334]: Waiting for error group managing segments... Dec 29 02:09:59.446248926 (562h40m59.317331961s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.448722897 (562h40m59.319805942s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.464395629 (562h40m59.335478674s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.488057194 (562h40m59.359140229s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.508989291 (562h40m59.380072336s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.533455388 (562h40m59.404538423s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.549045133 (562h40m59.420128178s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.550443936 (562h40m59.421526981s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.554893479 (562h40m59.425976514s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.594078902 (562h40m59.465161937s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.649264192 (562h40m59.520347227s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.650091498 (562h40m59.521174533s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690062978 (562h40m59.561146023s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690082646 (562h40m59.561165681s) [seg_pos={0 0}, upload=493334]: Segment reached optimal threshold of 13 pieces. Dec 29 02:09:59.690253742 (562h40m59.561336777s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection Dec 29 02:09:59.690259893 (562h40m59.561342928s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690326821 (562h40m59.561409856s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection Dec 29 02:09:59.690350265 (562h40m59.56143331s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690483589 (562h40m59.561566634s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection Dec 29 02:09:59.690511442 (562h40m59.561594477s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690526480 (562h40m59.561609525s) [seg_pos={0 0}, upload=493334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:09:59.690618406 (562h40m59.561701441s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:09:59.690623505 (562h40m59.56170654s) [upload=493334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:09:59.984752543 (562h40m59.855835579s) [upload=493334]: Done uploading segments... Dec 29 02:09:59.984779735 (562h40m59.855862771s) [upload=493334]: Upload finished. err: <nil> Duration:* 6.547411035s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X................XX....X.....X......X...X....X.....X........] duration: 6.018562921s trace: https://jaeger.dev.storj.io/trace/6457a1a4845279ab FAIL (2025-12-29 02:17:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (562h40m57.020745103s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:09:57.402920664 (562h40m57.274003699s) [seg_pos={0 0}, upload=493334]: Begin upload segment... Dec 29 02:09:57.402971150 (562h40m57.274054185s) [seg_pos={0 0}, upload=493334]: Done begin upload segment. Dec 29 02:09:57.403010204 (562h40m57.274093239s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403021486 (562h40m57.274104531s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403158967 (562h40m57.274242012s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403570060 (562h40m57.274653095s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403681472 (562h40m57.274764507s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403859191 (562h40m57.274942227s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404011882 (562h40m57.275094917s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404109588 (562h40m57.275192623s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404360806 (562h40m57.275443841s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404605462 (562h40m57.275688497s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404840620 (562h40m57.275923655s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405103179 (562h40m57.276186215s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405201777 (562h40m57.276284812s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405343958 (562h40m57.276426993s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405587281 (562h40m57.276670316s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405920556 (562h40m57.277003602s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:59.145565768 (562h40m59.016648803s) [upload=493334]: Next returned nil segment Dec 29 02:09:59.145581808 (562h40m59.016664853s) [upload=493334]: Waiting for error group managing segments... Dec 29 02:09:59.446248926 (562h40m59.317331961s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.448722897 (562h40m59.319805942s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.464395629 (562h40m59.335478674s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.488057194 (562h40m59.359140229s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.508989291 (562h40m59.380072336s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.533455388 (562h40m59.404538423s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.549045133 (562h40m59.420128178s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.550443936 (562h40m59.421526981s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.554893479 (562h40m59.425976514s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.594078902 (562h40m59.465161937s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.649264192 (562h40m59.520347227s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.650091498 (562h40m59.521174533s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690062978 (562h40m59.561146023s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690082646 (562h40m59.561165681s) [seg_pos={0 0}, upload=493334]: Segment reached optimal threshold of 13 pieces. Dec 29 02:09:59.690253742 (562h40m59.561336777s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection Dec 29 02:09:59.690259893 (562h40m59.561342928s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690326821 (562h40m59.561409856s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection Dec 29 02:09:59.690350265 (562h40m59.56143331s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690483589 (562h40m59.561566634s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection Dec 29 02:09:59.690511442 (562h40m59.561594477s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690526480 (562h40m59.561609525s) [seg_pos={0 0}, upload=493334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:09:59.690618406 (562h40m59.561701441s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:09:59.690623505 (562h40m59.56170654s) [upload=493334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:09:59.984752543 (562h40m59.855835579s) [upload=493334]: Done uploading segments... Dec 29 02:09:59.984779735 (562h40m59.855862771s) [upload=493334]: Upload finished. err: <nil> Duration:* 6.018585566s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX................XX....X.....X......X...X....X.....X.......] duration: 5.750548019s trace: https://jaeger.dev.storj.io/trace/6457a1a4845279ab FAIL (2025-12-29 02:15:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (562h40m57.020745103s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:09:57.402920664 (562h40m57.274003699s) [seg_pos={0 0}, upload=493334]: Begin upload segment... Dec 29 02:09:57.402971150 (562h40m57.274054185s) [seg_pos={0 0}, upload=493334]: Done begin upload segment. Dec 29 02:09:57.403010204 (562h40m57.274093239s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403021486 (562h40m57.274104531s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403158967 (562h40m57.274242012s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403570060 (562h40m57.274653095s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403681472 (562h40m57.274764507s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403859191 (562h40m57.274942227s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404011882 (562h40m57.275094917s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404109588 (562h40m57.275192623s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404360806 (562h40m57.275443841s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404605462 (562h40m57.275688497s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404840620 (562h40m57.275923655s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405103179 (562h40m57.276186215s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405201777 (562h40m57.276284812s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405343958 (562h40m57.276426993s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405587281 (562h40m57.276670316s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405920556 (562h40m57.277003602s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:59.145565768 (562h40m59.016648803s) [upload=493334]: Next returned nil segment Dec 29 02:09:59.145581808 (562h40m59.016664853s) [upload=493334]: Waiting for error group managing segments... Dec 29 02:09:59.446248926 (562h40m59.317331961s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.448722897 (562h40m59.319805942s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.464395629 (562h40m59.335478674s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.488057194 (562h40m59.359140229s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.508989291 (562h40m59.380072336s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.533455388 (562h40m59.404538423s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.549045133 (562h40m59.420128178s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.550443936 (562h40m59.421526981s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.554893479 (562h40m59.425976514s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.594078902 (562h40m59.465161937s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.649264192 (562h40m59.520347227s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.650091498 (562h40m59.521174533s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690062978 (562h40m59.561146023s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690082646 (562h40m59.561165681s) [seg_pos={0 0}, upload=493334]: Segment reached optimal threshold of 13 pieces. Dec 29 02:09:59.690253742 (562h40m59.561336777s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection Dec 29 02:09:59.690259893 (562h40m59.561342928s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690326821 (562h40m59.561409856s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection Dec 29 02:09:59.690350265 (562h40m59.56143331s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690483589 (562h40m59.561566634s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection Dec 29 02:09:59.690511442 (562h40m59.561594477s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690526480 (562h40m59.561609525s) [seg_pos={0 0}, upload=493334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:09:59.690618406 (562h40m59.561701441s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:09:59.690623505 (562h40m59.56170654s) [upload=493334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:09:59.984752543 (562h40m59.855835579s) [upload=493334]: Done uploading segments... Dec 29 02:09:59.984779735 (562h40m59.855862771s) [upload=493334]: Upload finished. err: <nil> Duration:* 5.750578478s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.XX................XX....X.....X......X...X....X.....X......] duration: 5.935020477s trace: https://jaeger.dev.storj.io/trace/6457a1a4845279ab FAIL (2025-12-29 02:15:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (562h40m57.020745103s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:09:57.402920664 (562h40m57.274003699s) [seg_pos={0 0}, upload=493334]: Begin upload segment... Dec 29 02:09:57.402971150 (562h40m57.274054185s) [seg_pos={0 0}, upload=493334]: Done begin upload segment. Dec 29 02:09:57.403010204 (562h40m57.274093239s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403021486 (562h40m57.274104531s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403158967 (562h40m57.274242012s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403570060 (562h40m57.274653095s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403681472 (562h40m57.274764507s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403859191 (562h40m57.274942227s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404011882 (562h40m57.275094917s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404109588 (562h40m57.275192623s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404360806 (562h40m57.275443841s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404605462 (562h40m57.275688497s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404840620 (562h40m57.275923655s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405103179 (562h40m57.276186215s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405201777 (562h40m57.276284812s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405343958 (562h40m57.276426993s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405587281 (562h40m57.276670316s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405920556 (562h40m57.277003602s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:59.145565768 (562h40m59.016648803s) [upload=493334]: Next returned nil segment Dec 29 02:09:59.145581808 (562h40m59.016664853s) [upload=493334]: Waiting for error group managing segments... Dec 29 02:09:59.446248926 (562h40m59.317331961s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.448722897 (562h40m59.319805942s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.464395629 (562h40m59.335478674s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.488057194 (562h40m59.359140229s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.508989291 (562h40m59.380072336s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.533455388 (562h40m59.404538423s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.549045133 (562h40m59.420128178s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.550443936 (562h40m59.421526981s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.554893479 (562h40m59.425976514s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.594078902 (562h40m59.465161937s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.649264192 (562h40m59.520347227s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.650091498 (562h40m59.521174533s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690062978 (562h40m59.561146023s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690082646 (562h40m59.561165681s) [seg_pos={0 0}, upload=493334]: Segment reached optimal threshold of 13 pieces. Dec 29 02:09:59.690253742 (562h40m59.561336777s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection Dec 29 02:09:59.690259893 (562h40m59.561342928s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690326821 (562h40m59.561409856s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection Dec 29 02:09:59.690350265 (562h40m59.56143331s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690483589 (562h40m59.561566634s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection Dec 29 02:09:59.690511442 (562h40m59.561594477s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690526480 (562h40m59.561609525s) [seg_pos={0 0}, upload=493334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:09:59.690618406 (562h40m59.561701441s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:09:59.690623505 (562h40m59.56170654s) [upload=493334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:09:59.984752543 (562h40m59.855835579s) [upload=493334]: Done uploading segments... Dec 29 02:09:59.984779735 (562h40m59.855862771s) [upload=493334]: Upload finished. err: <nil> Duration:* 5.935048349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..XX................XX....X.....X......X...X....X.....X.....] duration: 5.835912554s trace: https://jaeger.dev.storj.io/trace/6457a1a4845279ab FAIL (2025-12-29 02:14:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (562h40m57.020745103s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:09:57.402920664 (562h40m57.274003699s) [seg_pos={0 0}, upload=493334]: Begin upload segment... Dec 29 02:09:57.402971150 (562h40m57.274054185s) [seg_pos={0 0}, upload=493334]: Done begin upload segment. Dec 29 02:09:57.403010204 (562h40m57.274093239s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403021486 (562h40m57.274104531s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403158967 (562h40m57.274242012s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403570060 (562h40m57.274653095s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403681472 (562h40m57.274764507s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403859191 (562h40m57.274942227s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404011882 (562h40m57.275094917s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404109588 (562h40m57.275192623s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404360806 (562h40m57.275443841s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404605462 (562h40m57.275688497s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404840620 (562h40m57.275923655s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405103179 (562h40m57.276186215s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405201777 (562h40m57.276284812s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405343958 (562h40m57.276426993s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405587281 (562h40m57.276670316s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405920556 (562h40m57.277003602s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:59.145565768 (562h40m59.016648803s) [upload=493334]: Next returned nil segment Dec 29 02:09:59.145581808 (562h40m59.016664853s) [upload=493334]: Waiting for error group managing segments... Dec 29 02:09:59.446248926 (562h40m59.317331961s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.448722897 (562h40m59.319805942s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.464395629 (562h40m59.335478674s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.488057194 (562h40m59.359140229s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.508989291 (562h40m59.380072336s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.533455388 (562h40m59.404538423s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.549045133 (562h40m59.420128178s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.550443936 (562h40m59.421526981s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.554893479 (562h40m59.425976514s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.594078902 (562h40m59.465161937s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.649264192 (562h40m59.520347227s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.650091498 (562h40m59.521174533s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690062978 (562h40m59.561146023s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690082646 (562h40m59.561165681s) [seg_pos={0 0}, upload=493334]: Segment reached optimal threshold of 13 pieces. Dec 29 02:09:59.690253742 (562h40m59.561336777s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection Dec 29 02:09:59.690259893 (562h40m59.561342928s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690326821 (562h40m59.561409856s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection Dec 29 02:09:59.690350265 (562h40m59.56143331s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690483589 (562h40m59.561566634s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection Dec 29 02:09:59.690511442 (562h40m59.561594477s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690526480 (562h40m59.561609525s) [seg_pos={0 0}, upload=493334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:09:59.690618406 (562h40m59.561701441s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:09:59.690623505 (562h40m59.56170654s) [upload=493334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:09:59.984752543 (562h40m59.855835579s) [upload=493334]: Done uploading segments... Dec 29 02:09:59.984779735 (562h40m59.855862771s) [upload=493334]: Upload finished. err: <nil> Duration:* 5.835953331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..XX................XX....X.....X......X...X....X.....X....] duration: 6.769438182s trace: https://jaeger.dev.storj.io/trace/6457a1a4845279ab FAIL (2025-12-29 02:12:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (562h40m57.020745103s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:09:57.402920664 (562h40m57.274003699s) [seg_pos={0 0}, upload=493334]: Begin upload segment... Dec 29 02:09:57.402971150 (562h40m57.274054185s) [seg_pos={0 0}, upload=493334]: Done begin upload segment. Dec 29 02:09:57.403010204 (562h40m57.274093239s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403021486 (562h40m57.274104531s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403158967 (562h40m57.274242012s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403570060 (562h40m57.274653095s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403681472 (562h40m57.274764507s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403859191 (562h40m57.274942227s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404011882 (562h40m57.275094917s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404109588 (562h40m57.275192623s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404360806 (562h40m57.275443841s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404605462 (562h40m57.275688497s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404840620 (562h40m57.275923655s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405103179 (562h40m57.276186215s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405201777 (562h40m57.276284812s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405343958 (562h40m57.276426993s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405587281 (562h40m57.276670316s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405920556 (562h40m57.277003602s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:59.145565768 (562h40m59.016648803s) [upload=493334]: Next returned nil segment Dec 29 02:09:59.145581808 (562h40m59.016664853s) [upload=493334]: Waiting for error group managing segments... Dec 29 02:09:59.446248926 (562h40m59.317331961s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.448722897 (562h40m59.319805942s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.464395629 (562h40m59.335478674s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.488057194 (562h40m59.359140229s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.508989291 (562h40m59.380072336s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.533455388 (562h40m59.404538423s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.549045133 (562h40m59.420128178s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.550443936 (562h40m59.421526981s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.554893479 (562h40m59.425976514s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.594078902 (562h40m59.465161937s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.649264192 (562h40m59.520347227s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.650091498 (562h40m59.521174533s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690062978 (562h40m59.561146023s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690082646 (562h40m59.561165681s) [seg_pos={0 0}, upload=493334]: Segment reached optimal threshold of 13 pieces. Dec 29 02:09:59.690253742 (562h40m59.561336777s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection Dec 29 02:09:59.690259893 (562h40m59.561342928s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690326821 (562h40m59.561409856s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection Dec 29 02:09:59.690350265 (562h40m59.56143331s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690483589 (562h40m59.561566634s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection Dec 29 02:09:59.690511442 (562h40m59.561594477s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690526480 (562h40m59.561609525s) [seg_pos={0 0}, upload=493334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:09:59.690618406 (562h40m59.561701441s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:09:59.690623505 (562h40m59.56170654s) [upload=493334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:09:59.984752543 (562h40m59.855835579s) [upload=493334]: Done uploading segments... Dec 29 02:09:59.984779735 (562h40m59.855862771s) [upload=493334]: Upload finished. err: <nil> Duration:* 6.769466916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX..XX................XX....X.....X......X...X....X.....X...] duration: 17.254158599s trace: https://jaeger.dev.storj.io/trace/6457a1a4845279ab FAIL (2025-12-29 02:12:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (562h40m57.020745103s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:09:57.402920664 (562h40m57.274003699s) [seg_pos={0 0}, upload=493334]: Begin upload segment... Dec 29 02:09:57.402971150 (562h40m57.274054185s) [seg_pos={0 0}, upload=493334]: Done begin upload segment. Dec 29 02:09:57.403010204 (562h40m57.274093239s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403021486 (562h40m57.274104531s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403158967 (562h40m57.274242012s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403570060 (562h40m57.274653095s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403681472 (562h40m57.274764507s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403859191 (562h40m57.274942227s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404011882 (562h40m57.275094917s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404109588 (562h40m57.275192623s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404360806 (562h40m57.275443841s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404605462 (562h40m57.275688497s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404840620 (562h40m57.275923655s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405103179 (562h40m57.276186215s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405201777 (562h40m57.276284812s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405343958 (562h40m57.276426993s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405587281 (562h40m57.276670316s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405920556 (562h40m57.277003602s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:59.145565768 (562h40m59.016648803s) [upload=493334]: Next returned nil segment Dec 29 02:09:59.145581808 (562h40m59.016664853s) [upload=493334]: Waiting for error group managing segments... Dec 29 02:09:59.446248926 (562h40m59.317331961s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.448722897 (562h40m59.319805942s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.464395629 (562h40m59.335478674s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.488057194 (562h40m59.359140229s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.508989291 (562h40m59.380072336s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.533455388 (562h40m59.404538423s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.549045133 (562h40m59.420128178s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.550443936 (562h40m59.421526981s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.554893479 (562h40m59.425976514s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.594078902 (562h40m59.465161937s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.649264192 (562h40m59.520347227s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.650091498 (562h40m59.521174533s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690062978 (562h40m59.561146023s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690082646 (562h40m59.561165681s) [seg_pos={0 0}, upload=493334]: Segment reached optimal threshold of 13 pieces. Dec 29 02:09:59.690253742 (562h40m59.561336777s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection Dec 29 02:09:59.690259893 (562h40m59.561342928s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690326821 (562h40m59.561409856s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection Dec 29 02:09:59.690350265 (562h40m59.56143331s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690483589 (562h40m59.561566634s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection Dec 29 02:09:59.690511442 (562h40m59.561594477s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690526480 (562h40m59.561609525s) [seg_pos={0 0}, upload=493334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:09:59.690618406 (562h40m59.561701441s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:09:59.690623505 (562h40m59.56170654s) [upload=493334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:09:59.984752543 (562h40m59.855835579s) [upload=493334]: Done uploading segments... Dec 29 02:09:59.984779735 (562h40m59.855862771s) [upload=493334]: Upload finished. err: <nil> Duration:* 17.254199597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXX..XX................XX....X.....X......X...X....X.....X..] duration: 8.010969116s trace: https://jaeger.dev.storj.io/trace/6457a1a4845279ab FAIL (2025-12-29 02:11:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (562h40m57.020745103s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:09:57.402920664 (562h40m57.274003699s) [seg_pos={0 0}, upload=493334]: Begin upload segment... Dec 29 02:09:57.402971150 (562h40m57.274054185s) [seg_pos={0 0}, upload=493334]: Done begin upload segment. Dec 29 02:09:57.403010204 (562h40m57.274093239s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403021486 (562h40m57.274104531s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403158967 (562h40m57.274242012s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403570060 (562h40m57.274653095s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403681472 (562h40m57.274764507s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403859191 (562h40m57.274942227s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404011882 (562h40m57.275094917s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404109588 (562h40m57.275192623s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404360806 (562h40m57.275443841s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404605462 (562h40m57.275688497s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404840620 (562h40m57.275923655s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405103179 (562h40m57.276186215s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405201777 (562h40m57.276284812s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405343958 (562h40m57.276426993s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405587281 (562h40m57.276670316s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405920556 (562h40m57.277003602s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:59.145565768 (562h40m59.016648803s) [upload=493334]: Next returned nil segment Dec 29 02:09:59.145581808 (562h40m59.016664853s) [upload=493334]: Waiting for error group managing segments... Dec 29 02:09:59.446248926 (562h40m59.317331961s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.448722897 (562h40m59.319805942s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.464395629 (562h40m59.335478674s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.488057194 (562h40m59.359140229s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.508989291 (562h40m59.380072336s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.533455388 (562h40m59.404538423s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.549045133 (562h40m59.420128178s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.550443936 (562h40m59.421526981s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.554893479 (562h40m59.425976514s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.594078902 (562h40m59.465161937s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.649264192 (562h40m59.520347227s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.650091498 (562h40m59.521174533s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690062978 (562h40m59.561146023s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690082646 (562h40m59.561165681s) [seg_pos={0 0}, upload=493334]: Segment reached optimal threshold of 13 pieces. Dec 29 02:09:59.690253742 (562h40m59.561336777s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection Dec 29 02:09:59.690259893 (562h40m59.561342928s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690326821 (562h40m59.561409856s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection Dec 29 02:09:59.690350265 (562h40m59.56143331s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690483589 (562h40m59.561566634s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection Dec 29 02:09:59.690511442 (562h40m59.561594477s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690526480 (562h40m59.561609525s) [seg_pos={0 0}, upload=493334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:09:59.690618406 (562h40m59.561701441s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:09:59.690623505 (562h40m59.56170654s) [upload=493334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:09:59.984752543 (562h40m59.855835579s) [upload=493334]: Done uploading segments... Dec 29 02:09:59.984779735 (562h40m59.855862771s) [upload=493334]: Upload finished. err: <nil> Duration:* 8.010991858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XXX..XX................XX....X.....X......X...X....X.....X.] duration: 5.801674628s trace: https://jaeger.dev.storj.io/trace/6457a1a4845279ab FAIL (2025-12-29 02:10:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (562h40m57.020745103s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:09:57.402920664 (562h40m57.274003699s) [seg_pos={0 0}, upload=493334]: Begin upload segment... Dec 29 02:09:57.402971150 (562h40m57.274054185s) [seg_pos={0 0}, upload=493334]: Done begin upload segment. Dec 29 02:09:57.403010204 (562h40m57.274093239s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403021486 (562h40m57.274104531s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403158967 (562h40m57.274242012s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403570060 (562h40m57.274653095s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403681472 (562h40m57.274764507s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403859191 (562h40m57.274942227s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404011882 (562h40m57.275094917s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404109588 (562h40m57.275192623s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404360806 (562h40m57.275443841s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404605462 (562h40m57.275688497s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404840620 (562h40m57.275923655s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405103179 (562h40m57.276186215s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405201777 (562h40m57.276284812s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405343958 (562h40m57.276426993s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405587281 (562h40m57.276670316s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405920556 (562h40m57.277003602s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:59.145565768 (562h40m59.016648803s) [upload=493334]: Next returned nil segment Dec 29 02:09:59.145581808 (562h40m59.016664853s) [upload=493334]: Waiting for error group managing segments... Dec 29 02:09:59.446248926 (562h40m59.317331961s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.448722897 (562h40m59.319805942s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.464395629 (562h40m59.335478674s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.488057194 (562h40m59.359140229s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.508989291 (562h40m59.380072336s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.533455388 (562h40m59.404538423s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.549045133 (562h40m59.420128178s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.550443936 (562h40m59.421526981s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.554893479 (562h40m59.425976514s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.594078902 (562h40m59.465161937s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.649264192 (562h40m59.520347227s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.650091498 (562h40m59.521174533s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690062978 (562h40m59.561146023s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690082646 (562h40m59.561165681s) [seg_pos={0 0}, upload=493334]: Segment reached optimal threshold of 13 pieces. Dec 29 02:09:59.690253742 (562h40m59.561336777s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection Dec 29 02:09:59.690259893 (562h40m59.561342928s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690326821 (562h40m59.561409856s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection Dec 29 02:09:59.690350265 (562h40m59.56143331s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690483589 (562h40m59.561566634s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection Dec 29 02:09:59.690511442 (562h40m59.561594477s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690526480 (562h40m59.561609525s) [seg_pos={0 0}, upload=493334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:09:59.690618406 (562h40m59.561701441s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:09:59.690623505 (562h40m59.56170654s) [upload=493334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:09:59.984752543 (562h40m59.855835579s) [upload=493334]: Done uploading segments... Dec 29 02:09:59.984779735 (562h40m59.855862771s) [upload=493334]: Upload finished. err: <nil> Duration:* 5.801690009s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XXX..XX................XX....X.....X......X...X....X.....X] duration: 30.002730407s trace: https://jaeger.dev.storj.io/trace/6457a1a4845279ab FAIL (2025-12-29 02:10:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (562h40m57.020745103s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:09:57.402920664 (562h40m57.274003699s) [seg_pos={0 0}, upload=493334]: Begin upload segment... Dec 29 02:09:57.402971150 (562h40m57.274054185s) [seg_pos={0 0}, upload=493334]: Done begin upload segment. Dec 29 02:09:57.403010204 (562h40m57.274093239s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403021486 (562h40m57.274104531s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403158967 (562h40m57.274242012s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403570060 (562h40m57.274653095s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403681472 (562h40m57.274764507s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.403859191 (562h40m57.274942227s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404011882 (562h40m57.275094917s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404109588 (562h40m57.275192623s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404360806 (562h40m57.275443841s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404605462 (562h40m57.275688497s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.404840620 (562h40m57.275923655s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405103179 (562h40m57.276186215s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405201777 (562h40m57.276284812s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405343958 (562h40m57.276426993s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405587281 (562h40m57.276670316s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:57.405920556 (562h40m57.277003602s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Uploading piece... Dec 29 02:09:59.145565768 (562h40m59.016648803s) [upload=493334]: Next returned nil segment Dec 29 02:09:59.145581808 (562h40m59.016664853s) [upload=493334]: Waiting for error group managing segments... Dec 29 02:09:59.446248926 (562h40m59.317331961s) [noise=true , address=145.79.227.46:4603 , piece_id=GDQYBQDDSWGCE3ESASBGODUI7AI65BUGJ6Y5MF25RKCQD2NXW4WA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.448722897 (562h40m59.319805942s) [noise=true , address=145.79.227.36:4518 , piece_id=IDC6DU62LONKGPD4VBHQEKHIOICC6JVOPPAPH4GYZWXFX2IVSBXQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.464395629 (562h40m59.335478674s) [noise=true , address=145.79.227.15:4875 , piece_id=SA3LKXFAUOCFBTY4KGKIGIGKJJJJF2GDXYFMA6Q2IG6KJVXJPTJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.488057194 (562h40m59.359140229s) [noise=true , address=145.79.227.52:4446 , piece_id=VLHUDLQFXHXWGS4DJGUVKK4J5VHRSXLEFC5HNSX2OBQKNIZA7BPA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.508989291 (562h40m59.380072336s) [noise=true , address=145.79.227.45:4231 , piece_id=ONL3JP4GEWQFPUWXKQEM4JXXS35UCMPLREXFLUPXWGQRRBII5FAA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.533455388 (562h40m59.404538423s) [noise=true , address=145.79.227.8:4330 , piece_id=IMA4LNVWZXXTU5ABZ6PICQARSCLNG7GW35BH4USGCLKVHYQQ6N2Q, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.549045133 (562h40m59.420128178s) [noise=true , address=145.79.227.4:4187 , piece_id=3FVOINKIRPHPSLEC7YDQKO55ABWBUPTD6U4OBPH5J7OXO56I7ZWA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.550443936 (562h40m59.421526981s) [noise=true , address=145.79.227.2:4113 , piece_id=Q3RBVASBX2TEB3IGTRIRDCBW6SZBO254UUJ7C2EBPWUXQGYTNGNA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.554893479 (562h40m59.425976514s) [noise=true , address=145.79.227.37:4565 , piece_id=LNABOZKGZW2GOJHSTQXE5TSTJ66ATFKYFNLGEXJPDF5XYPX7TWQA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.594078902 (562h40m59.465161937s) [noise=true , address=77.90.31.11:4553 , piece_id=7HP4XNIOTZHGRRJL4SY73FJCFSXDTILFZKPSANMPIALIT7IV6TYA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.649264192 (562h40m59.520347227s) [noise=true , address=77.90.31.21:4877 , piece_id=CVJW7PG54C5ZP5TDMS7MAU374GD37X26FNNZ2PMVBALENM54NWJA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.650091498 (562h40m59.521174533s) [noise=true , address=45.91.116.10:4083 , piece_id=6QSGC2G6CEDM42P72YXJY3QZB5RFDOHOBXQZTEE5MBQ6JP3DRIJQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690062978 (562h40m59.561146023s) [noise=true , address=77.90.31.17:4773 , piece_id=Z3YEIH3CXMHBBATPKTFNJS3G7RXT2UUWD6MJXRHB7MR3K5UY2PLA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: <nil> Dec 29 02:09:59.690082646 (562h40m59.561165681s) [seg_pos={0 0}, upload=493334]: Segment reached optimal threshold of 13 pieces. Dec 29 02:09:59.690253742 (562h40m59.561336777s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37040->45.91.116.6:4973: use of closed network connection Dec 29 02:09:59.690259893 (562h40m59.561342928s) [noise=true , address=45.91.116.6:4973 , piece_id=CH6274WXMCSAXDQHMNJEPIULPGK4ILJ32RDIKARF2I44ZRANP63A, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690326821 (562h40m59.561409856s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52644->45.91.116.14:4190: use of closed network connection Dec 29 02:09:59.690350265 (562h40m59.56143331s) [noise=true , address=45.91.116.14:4190 , piece_id=NOCPJF4KDAOQJQRVEEWDXXA6LB3UTMKOLKX47RCYTBXF3BNECESQ, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690483589 (562h40m59.561566634s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g): protocol: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52882->145.79.227.13:4804: use of closed network connection Dec 29 02:09:59.690511442 (562h40m59.561594477s) [noise=true , address=145.79.227.13:4804 , piece_id=KR65AZVI72ZVQOVUYLEJI4X4T57M3FVEPXHVGDDZGBHYPQB47RKA, seg_pos={0 0}, upload=493334]: Long tail cancellation due to optimal threshold Dec 29 02:09:59.690526480 (562h40m59.561609525s) [seg_pos={0 0}, upload=493334]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:09:59.690618406 (562h40m59.561701441s) [upload=493334]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:09:59.690623505 (562h40m59.56170654s) [upload=493334]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:09:59.984752543 (562h40m59.855835579s) [upload=493334]: Done uploading segments... Dec 29 02:09:59.984779735 (562h40m59.855862771s) [upload=493334]: Upload finished. err: <nil> Duration:* 30.00275329s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...XXX..XX................XX....X.....X......X...X....X.....] duration: 6.405409467s trace: https://jaeger.dev.storj.io/trace/1d946b76eaf9cd0b FAIL (2025-12-29 02:09:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA) context deadline exceeded ======LOGS====== Dec 29 02:05:43.674378070 (562h36m43.545461116s) [upload=493268]: Starting upload Dec 29 02:05:43.674484784 (562h36m43.545567829s) [upload=493268]: Done starting upload Dec 29 02:05:43.674681719 (562h36m43.545764754s) [upload=493268]: Uploading segments... Dec 29 02:05:43.675209634 (562h36m43.546292679s) [upload=493268]: Got next segment. Inline: false Dec 29 02:05:43.675286961 (562h36m43.546369996s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:05:43.675294075 (562h36m43.54637712s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:05:43.920110235 (562h36m43.79119328s) [seg_pos={0 0}, upload=493268]: Begin upload segment... Dec 29 02:05:43.920164408 (562h36m43.791247454s) [seg_pos={0 0}, upload=493268]: Done begin upload segment. Dec 29 02:05:43.920200227 (562h36m43.791283272s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.921487256 (562h36m43.792570301s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.922316235 (562h36m43.79339927s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923015016 (562h36m43.794098051s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923452820 (562h36m43.794535865s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924048875 (562h36m43.79513192s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924418178 (562h36m43.795501223s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925022719 (562h36m43.796105754s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925412461 (562h36m43.796495496s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.926840149 (562h36m43.797923195s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927261462 (562h36m43.798344497s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927757166 (562h36m43.798840211s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.928924448 (562h36m43.800007484s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.930299587 (562h36m43.801382623s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.931538787 (562h36m43.802621822s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.932875471 (562h36m43.803958516s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:45.604910193 (562h36m45.475993228s) [upload=493268]: Next returned nil segment Dec 29 02:05:45.604921885 (562h36m45.47600492s) [upload=493268]: Waiting for error group managing segments... Dec 29 02:05:45.919563633 (562h36m45.790646678s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.933885923 (562h36m45.804968968s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.935261483 (562h36m45.806344538s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.940436266 (562h36m45.811519301s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950301842 (562h36m45.821384877s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950377265 (562h36m45.82146031s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951022554 (562h36m45.822105599s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951257632 (562h36m45.822340677s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951924852 (562h36m45.823007887s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.957340524 (562h36m45.828423569s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.103325488 (562h36m45.974408523s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.124583115 (562h36m45.99566615s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322434829 (562h36m46.193517874s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322460408 (562h36m46.193543453s) [seg_pos={0 0}, upload=493268]: Segment reached optimal threshold of 13 pieces. Dec 29 02:05:46.322798140 (562h36m46.193881185s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322870217 (562h36m46.193953262s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.322917799 (562h36m46.194000844s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322928990 (562h36m46.194012035s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323154128 (562h36m46.194237173s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.323161342 (562h36m46.194244377s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323178905 (562h36m46.19426194s) [seg_pos={0 0}, upload=493268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:05:46.323287762 (562h36m46.194370798s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:05:46.323293543 (562h36m46.194376578s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:05:46.609304736 (562h36m46.480387771s) [upload=493268]: Done uploading segments... Dec 29 02:05:46.609324854 (562h36m46.480407889s) [upload=493268]: Upload finished. err: <nil> Duration: 6.405435977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....XXX..XX................XX....X.....X......X...X....X....] duration: 6.135892566s trace: https://jaeger.dev.storj.io/trace/1d946b76eaf9cd0b FAIL (2025-12-29 02:08:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA) context deadline exceeded ======LOGS====== Dec 29 02:05:43.674378070 (562h36m43.545461116s) [upload=493268]: Starting upload Dec 29 02:05:43.674484784 (562h36m43.545567829s) [upload=493268]: Done starting upload Dec 29 02:05:43.674681719 (562h36m43.545764754s) [upload=493268]: Uploading segments... Dec 29 02:05:43.675209634 (562h36m43.546292679s) [upload=493268]: Got next segment. Inline: false Dec 29 02:05:43.675286961 (562h36m43.546369996s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:05:43.675294075 (562h36m43.54637712s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:05:43.920110235 (562h36m43.79119328s) [seg_pos={0 0}, upload=493268]: Begin upload segment... Dec 29 02:05:43.920164408 (562h36m43.791247454s) [seg_pos={0 0}, upload=493268]: Done begin upload segment. Dec 29 02:05:43.920200227 (562h36m43.791283272s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.921487256 (562h36m43.792570301s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.922316235 (562h36m43.79339927s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923015016 (562h36m43.794098051s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923452820 (562h36m43.794535865s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924048875 (562h36m43.79513192s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924418178 (562h36m43.795501223s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925022719 (562h36m43.796105754s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925412461 (562h36m43.796495496s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.926840149 (562h36m43.797923195s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927261462 (562h36m43.798344497s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927757166 (562h36m43.798840211s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.928924448 (562h36m43.800007484s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.930299587 (562h36m43.801382623s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.931538787 (562h36m43.802621822s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.932875471 (562h36m43.803958516s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:45.604910193 (562h36m45.475993228s) [upload=493268]: Next returned nil segment Dec 29 02:05:45.604921885 (562h36m45.47600492s) [upload=493268]: Waiting for error group managing segments... Dec 29 02:05:45.919563633 (562h36m45.790646678s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.933885923 (562h36m45.804968968s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.935261483 (562h36m45.806344538s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.940436266 (562h36m45.811519301s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950301842 (562h36m45.821384877s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950377265 (562h36m45.82146031s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951022554 (562h36m45.822105599s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951257632 (562h36m45.822340677s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951924852 (562h36m45.823007887s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.957340524 (562h36m45.828423569s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.103325488 (562h36m45.974408523s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.124583115 (562h36m45.99566615s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322434829 (562h36m46.193517874s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322460408 (562h36m46.193543453s) [seg_pos={0 0}, upload=493268]: Segment reached optimal threshold of 13 pieces. Dec 29 02:05:46.322798140 (562h36m46.193881185s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322870217 (562h36m46.193953262s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.322917799 (562h36m46.194000844s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322928990 (562h36m46.194012035s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323154128 (562h36m46.194237173s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.323161342 (562h36m46.194244377s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323178905 (562h36m46.19426194s) [seg_pos={0 0}, upload=493268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:05:46.323287762 (562h36m46.194370798s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:05:46.323293543 (562h36m46.194376578s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:05:46.609304736 (562h36m46.480387771s) [upload=493268]: Done uploading segments... Dec 29 02:05:46.609324854 (562h36m46.480407889s) [upload=493268]: Upload finished. err: <nil> Duration: 6.135925027s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X....XXX..XX................XX....X.....X......X...X....X...] duration: 6.100378409s trace: https://jaeger.dev.storj.io/trace/1d946b76eaf9cd0b FAIL (2025-12-29 02:07:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA) context deadline exceeded ======LOGS====== Dec 29 02:05:43.674378070 (562h36m43.545461116s) [upload=493268]: Starting upload Dec 29 02:05:43.674484784 (562h36m43.545567829s) [upload=493268]: Done starting upload Dec 29 02:05:43.674681719 (562h36m43.545764754s) [upload=493268]: Uploading segments... Dec 29 02:05:43.675209634 (562h36m43.546292679s) [upload=493268]: Got next segment. Inline: false Dec 29 02:05:43.675286961 (562h36m43.546369996s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:05:43.675294075 (562h36m43.54637712s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:05:43.920110235 (562h36m43.79119328s) [seg_pos={0 0}, upload=493268]: Begin upload segment... Dec 29 02:05:43.920164408 (562h36m43.791247454s) [seg_pos={0 0}, upload=493268]: Done begin upload segment. Dec 29 02:05:43.920200227 (562h36m43.791283272s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.921487256 (562h36m43.792570301s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.922316235 (562h36m43.79339927s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923015016 (562h36m43.794098051s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923452820 (562h36m43.794535865s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924048875 (562h36m43.79513192s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924418178 (562h36m43.795501223s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925022719 (562h36m43.796105754s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925412461 (562h36m43.796495496s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.926840149 (562h36m43.797923195s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927261462 (562h36m43.798344497s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927757166 (562h36m43.798840211s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.928924448 (562h36m43.800007484s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.930299587 (562h36m43.801382623s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.931538787 (562h36m43.802621822s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.932875471 (562h36m43.803958516s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:45.604910193 (562h36m45.475993228s) [upload=493268]: Next returned nil segment Dec 29 02:05:45.604921885 (562h36m45.47600492s) [upload=493268]: Waiting for error group managing segments... Dec 29 02:05:45.919563633 (562h36m45.790646678s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.933885923 (562h36m45.804968968s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.935261483 (562h36m45.806344538s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.940436266 (562h36m45.811519301s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950301842 (562h36m45.821384877s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950377265 (562h36m45.82146031s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951022554 (562h36m45.822105599s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951257632 (562h36m45.822340677s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951924852 (562h36m45.823007887s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.957340524 (562h36m45.828423569s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.103325488 (562h36m45.974408523s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.124583115 (562h36m45.99566615s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322434829 (562h36m46.193517874s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322460408 (562h36m46.193543453s) [seg_pos={0 0}, upload=493268]: Segment reached optimal threshold of 13 pieces. Dec 29 02:05:46.322798140 (562h36m46.193881185s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322870217 (562h36m46.193953262s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.322917799 (562h36m46.194000844s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322928990 (562h36m46.194012035s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323154128 (562h36m46.194237173s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.323161342 (562h36m46.194244377s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323178905 (562h36m46.19426194s) [seg_pos={0 0}, upload=493268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:05:46.323287762 (562h36m46.194370798s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:05:46.323293543 (562h36m46.194376578s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:05:46.609304736 (562h36m46.480387771s) [upload=493268]: Done uploading segments... Dec 29 02:05:46.609324854 (562h36m46.480407889s) [upload=493268]: Upload finished. err: <nil> Duration: 6.100420439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X....XXX..XX................XX....X.....X......X...X....X..] duration: 6.74854976s trace: https://jaeger.dev.storj.io/trace/1d946b76eaf9cd0b FAIL (2025-12-29 02:06:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA) context deadline exceeded ======LOGS====== Dec 29 02:05:43.674378070 (562h36m43.545461116s) [upload=493268]: Starting upload Dec 29 02:05:43.674484784 (562h36m43.545567829s) [upload=493268]: Done starting upload Dec 29 02:05:43.674681719 (562h36m43.545764754s) [upload=493268]: Uploading segments... Dec 29 02:05:43.675209634 (562h36m43.546292679s) [upload=493268]: Got next segment. Inline: false Dec 29 02:05:43.675286961 (562h36m43.546369996s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:05:43.675294075 (562h36m43.54637712s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:05:43.920110235 (562h36m43.79119328s) [seg_pos={0 0}, upload=493268]: Begin upload segment... Dec 29 02:05:43.920164408 (562h36m43.791247454s) [seg_pos={0 0}, upload=493268]: Done begin upload segment. Dec 29 02:05:43.920200227 (562h36m43.791283272s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.921487256 (562h36m43.792570301s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.922316235 (562h36m43.79339927s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923015016 (562h36m43.794098051s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923452820 (562h36m43.794535865s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924048875 (562h36m43.79513192s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924418178 (562h36m43.795501223s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925022719 (562h36m43.796105754s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925412461 (562h36m43.796495496s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.926840149 (562h36m43.797923195s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927261462 (562h36m43.798344497s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927757166 (562h36m43.798840211s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.928924448 (562h36m43.800007484s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.930299587 (562h36m43.801382623s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.931538787 (562h36m43.802621822s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.932875471 (562h36m43.803958516s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:45.604910193 (562h36m45.475993228s) [upload=493268]: Next returned nil segment Dec 29 02:05:45.604921885 (562h36m45.47600492s) [upload=493268]: Waiting for error group managing segments... Dec 29 02:05:45.919563633 (562h36m45.790646678s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.933885923 (562h36m45.804968968s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.935261483 (562h36m45.806344538s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.940436266 (562h36m45.811519301s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950301842 (562h36m45.821384877s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950377265 (562h36m45.82146031s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951022554 (562h36m45.822105599s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951257632 (562h36m45.822340677s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951924852 (562h36m45.823007887s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.957340524 (562h36m45.828423569s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.103325488 (562h36m45.974408523s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.124583115 (562h36m45.99566615s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322434829 (562h36m46.193517874s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322460408 (562h36m46.193543453s) [seg_pos={0 0}, upload=493268]: Segment reached optimal threshold of 13 pieces. Dec 29 02:05:46.322798140 (562h36m46.193881185s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322870217 (562h36m46.193953262s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.322917799 (562h36m46.194000844s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322928990 (562h36m46.194012035s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323154128 (562h36m46.194237173s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.323161342 (562h36m46.194244377s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323178905 (562h36m46.19426194s) [seg_pos={0 0}, upload=493268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:05:46.323287762 (562h36m46.194370798s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:05:46.323293543 (562h36m46.194376578s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:05:46.609304736 (562h36m46.480387771s) [upload=493268]: Done uploading segments... Dec 29 02:05:46.609324854 (562h36m46.480407889s) [upload=493268]: Upload finished. err: <nil> Duration: 6.748577784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X....XXX..XX................XX....X.....X......X...X....X.] duration: 6.207369484s trace: https://jaeger.dev.storj.io/trace/1d946b76eaf9cd0b FAIL (2025-12-29 02:06:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA) context deadline exceeded ======LOGS====== Dec 29 02:05:43.674378070 (562h36m43.545461116s) [upload=493268]: Starting upload Dec 29 02:05:43.674484784 (562h36m43.545567829s) [upload=493268]: Done starting upload Dec 29 02:05:43.674681719 (562h36m43.545764754s) [upload=493268]: Uploading segments... Dec 29 02:05:43.675209634 (562h36m43.546292679s) [upload=493268]: Got next segment. Inline: false Dec 29 02:05:43.675286961 (562h36m43.546369996s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:05:43.675294075 (562h36m43.54637712s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:05:43.920110235 (562h36m43.79119328s) [seg_pos={0 0}, upload=493268]: Begin upload segment... Dec 29 02:05:43.920164408 (562h36m43.791247454s) [seg_pos={0 0}, upload=493268]: Done begin upload segment. Dec 29 02:05:43.920200227 (562h36m43.791283272s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.921487256 (562h36m43.792570301s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.922316235 (562h36m43.79339927s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923015016 (562h36m43.794098051s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923452820 (562h36m43.794535865s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924048875 (562h36m43.79513192s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924418178 (562h36m43.795501223s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925022719 (562h36m43.796105754s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925412461 (562h36m43.796495496s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.926840149 (562h36m43.797923195s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927261462 (562h36m43.798344497s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927757166 (562h36m43.798840211s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.928924448 (562h36m43.800007484s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.930299587 (562h36m43.801382623s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.931538787 (562h36m43.802621822s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.932875471 (562h36m43.803958516s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:45.604910193 (562h36m45.475993228s) [upload=493268]: Next returned nil segment Dec 29 02:05:45.604921885 (562h36m45.47600492s) [upload=493268]: Waiting for error group managing segments... Dec 29 02:05:45.919563633 (562h36m45.790646678s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.933885923 (562h36m45.804968968s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.935261483 (562h36m45.806344538s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.940436266 (562h36m45.811519301s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950301842 (562h36m45.821384877s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950377265 (562h36m45.82146031s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951022554 (562h36m45.822105599s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951257632 (562h36m45.822340677s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951924852 (562h36m45.823007887s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.957340524 (562h36m45.828423569s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.103325488 (562h36m45.974408523s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.124583115 (562h36m45.99566615s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322434829 (562h36m46.193517874s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322460408 (562h36m46.193543453s) [seg_pos={0 0}, upload=493268]: Segment reached optimal threshold of 13 pieces. Dec 29 02:05:46.322798140 (562h36m46.193881185s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322870217 (562h36m46.193953262s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.322917799 (562h36m46.194000844s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322928990 (562h36m46.194012035s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323154128 (562h36m46.194237173s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.323161342 (562h36m46.194244377s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323178905 (562h36m46.19426194s) [seg_pos={0 0}, upload=493268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:05:46.323287762 (562h36m46.194370798s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:05:46.323293543 (562h36m46.194376578s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:05:46.609304736 (562h36m46.480387771s) [upload=493268]: Done uploading segments... Dec 29 02:05:46.609324854 (562h36m46.480407889s) [upload=493268]: Upload finished. err: <nil> Duration: 6.207395985s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X....XXX..XX................XX....X.....X......X...X....X] duration: 30.002445339s trace: https://jaeger.dev.storj.io/trace/1d946b76eaf9cd0b FAIL (2025-12-29 02:06:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA) context deadline exceeded ======LOGS====== Dec 29 02:05:43.674378070 (562h36m43.545461116s) [upload=493268]: Starting upload Dec 29 02:05:43.674484784 (562h36m43.545567829s) [upload=493268]: Done starting upload Dec 29 02:05:43.674681719 (562h36m43.545764754s) [upload=493268]: Uploading segments... Dec 29 02:05:43.675209634 (562h36m43.546292679s) [upload=493268]: Got next segment. Inline: false Dec 29 02:05:43.675286961 (562h36m43.546369996s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:05:43.675294075 (562h36m43.54637712s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:05:43.920110235 (562h36m43.79119328s) [seg_pos={0 0}, upload=493268]: Begin upload segment... Dec 29 02:05:43.920164408 (562h36m43.791247454s) [seg_pos={0 0}, upload=493268]: Done begin upload segment. Dec 29 02:05:43.920200227 (562h36m43.791283272s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.921487256 (562h36m43.792570301s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.922316235 (562h36m43.79339927s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923015016 (562h36m43.794098051s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.923452820 (562h36m43.794535865s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924048875 (562h36m43.79513192s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.924418178 (562h36m43.795501223s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925022719 (562h36m43.796105754s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.925412461 (562h36m43.796495496s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.926840149 (562h36m43.797923195s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927261462 (562h36m43.798344497s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.927757166 (562h36m43.798840211s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.928924448 (562h36m43.800007484s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.930299587 (562h36m43.801382623s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.931538787 (562h36m43.802621822s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:43.932875471 (562h36m43.803958516s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Uploading piece... Dec 29 02:05:45.604910193 (562h36m45.475993228s) [upload=493268]: Next returned nil segment Dec 29 02:05:45.604921885 (562h36m45.47600492s) [upload=493268]: Waiting for error group managing segments... Dec 29 02:05:45.919563633 (562h36m45.790646678s) [noise=true , address=145.79.227.17:4972 , piece_id=RIRPZYVS57KYECVJQRJAZFOQWAL3GKED3VHIPZYJWFRZQOCWASEA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.933885923 (562h36m45.804968968s) [noise=true , address=145.79.227.52:4447 , piece_id=GHKEJRMJAATYB5W5GUYKEFNJNBYDP5GMFBSJ73WNXWQUL53GSQSQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.935261483 (562h36m45.806344538s) [noise=true , address=145.79.227.6:4258 , piece_id=JFLLH4KYASE2UHEYFQ23LP7TTFBJBJHWHXJDUQNXQ3A7TLYKEUXA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.940436266 (562h36m45.811519301s) [noise=true , address=145.79.227.14:4840 , piece_id=MIRHQDQGXRZA66GIMXABFSAMMVDYEHWXKYWBR7IHILZEYBFE7NTA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950301842 (562h36m45.821384877s) [noise=true , address=145.79.227.38:4977 , piece_id=H22CRTHUJZQGTBQLP4SXCGHQ6HXPKXJMURP7XAPYE6USJ3C7NQHQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.950377265 (562h36m45.82146031s) [noise=true , address=145.79.227.41:4085 , piece_id=HLCXLN7BYUM2ZEPHN573OXE7SJA7YEJYL53SSMKQZIORMYBXBUMA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951022554 (562h36m45.822105599s) [noise=true , address=145.79.227.4:4186 , piece_id=KWGDFVOID5MBZG4FYWJZLGIMUAFYOKCOGPNQWN4VBF337BVJZHOA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951257632 (562h36m45.822340677s) [noise=true , address=145.79.227.18:4007 , piece_id=BQJWOBYWVZH7VYRUTLQK2NOIWU5YSQOKJAE4GNCCZUO7FSNL4N5A, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.951924852 (562h36m45.823007887s) [noise=true , address=145.79.227.32:4835 , piece_id=UW4CFCIL7AYTMFM6PMHAYKBQSTPMDZ4RLNNQOLQL5O6SKE5BR7KQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:45.957340524 (562h36m45.828423569s) [noise=true , address=145.79.227.51:4411 , piece_id=XHNLD6UJAJMUB57T3ILVKD2MYPLNXC66DGODYKGCP7SXBROOX4AA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.103325488 (562h36m45.974408523s) [noise=true , address=77.90.31.17:4770 , piece_id=MR7PEHPC364A3J3CVTRT7HWGV2VOMXO43AG64NIKUOLO5U3KW3SA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.124583115 (562h36m45.99566615s) [noise=true , address=145.79.227.15:4874 , piece_id=DVJWSZBUC5YDBVCNT7VHDJ5MROYR2WWZHZ5LY5OOI6TBRQGKAXKA, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322434829 (562h36m46.193517874s) [noise=true , address=45.91.116.6:4973 , piece_id=WU33PE5Z5EVDH4XNPGCTN7T2F6WSUSFAGVV4SOYSBS6PMQJ6N2XQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: <nil> Dec 29 02:05:46.322460408 (562h36m46.193543453s) [seg_pos={0 0}, upload=493268]: Segment reached optimal threshold of 13 pieces. Dec 29 02:05:46.322798140 (562h36m46.193881185s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322870217 (562h36m46.193953262s) [noise=true , address=45.91.116.14:4191 , piece_id=Z7JYMD2EQOSEHSTS7D7KS6ZKBEBWPMWBC3E2JXYQIBBOACZFY2CQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.322917799 (562h36m46.194000844s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.322928990 (562h36m46.194012035s) [noise=true , address=145.79.227.34:4907 , piece_id=H72NFYLZ73NO2JVK7PQR6TMQO2M62JKWM6M43UODB7ITXVNHKVUQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323154128 (562h36m46.194237173s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 02:05:46.323161342 (562h36m46.194244377s) [noise=true , address=77.90.31.10:4520 , piece_id=37LO4MHGCQ77JTJSGIEETFM3DFP4HA5OR7PU6VMLC5RAJ26QIPQQ, seg_pos={0 0}, upload=493268]: Long tail cancellation due to optimal threshold Dec 29 02:05:46.323178905 (562h36m46.19426194s) [seg_pos={0 0}, upload=493268]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:05:46.323287762 (562h36m46.194370798s) [upload=493268]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:05:46.323293543 (562h36m46.194376578s) [upload=493268]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:05:46.609304736 (562h36m46.480387771s) [upload=493268]: Done uploading segments... Dec 29 02:05:46.609324854 (562h36m46.480407889s) [upload=493268]: Upload finished. err: <nil> Duration: 30.002470076s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X....XXX..XX................XX....X.....X......X...X....] duration: 5.876069716s trace: https://jaeger.dev.storj.io/trace/4d643c0096eb6966 FAIL (2025-12-29 02:04:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA) context deadline exceeded ======LOGS====== 9 02:02:34.282577299 (562h33m34.153660344s) [upload=493222]: Starting upload Dec 29 02:02:34.282654466 (562h33m34.153737501s) [upload=493222]: Done starting upload Dec 29 02:02:34.282739407 (562h33m34.153822452s) [upload=493222]: Uploading segments... Dec 29 02:02:34.282966710 (562h33m34.154049755s) [upload=493222]: Got next segment. Inline: false Dec 29 02:02:34.283001496 (562h33m34.154084532s) [upload=493222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:02:34.283005694 (562h33m34.154088739s) [upload=493222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:02:34.527560786 (562h33m34.398643831s) [seg_pos={0 0}, upload=493222]: Begin upload segment... Dec 29 02:02:34.527626991 (562h33m34.398710036s) [seg_pos={0 0}, upload=493222]: Done begin upload segment. Dec 29 02:02:34.527648643 (562h33m34.398731678s) [noise=true , address=45.91.116.8:4045 , piece_id=NRJYAMG75VAZAK7A3RN4JUW42CHFEHWA72MUUXFHGRXGGY67LCWA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.527721772 (562h33m34.398804817s) [noise=true , address=145.79.227.11:4731 , piece_id=KCDJJQZX6AXJL3DDN662JZIFZ5VOVAQ53EZE7VYA5TABAFZE677Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528050358 (562h33m34.399133393s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528330812 (562h33m34.399413847s) [noise=true , address=45.91.116.13:4156 , piece_id=J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528610094 (562h33m34.399693129s) [noise=true , address=145.79.227.45:4230 , piece_id=TJYP2EE5H5IYRNYVJ33CWMGGQ5XCYIQBT7ZABBZCEGAU4PKAKB3A, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528872794 (562h33m34.399955829s) [noise=true , address=145.79.227.8:4330 , piece_id=RWLDKHK3TEKQJLTBGG3OCBIHQLN74VQXHFB3C5YFKIIPAQDLOOMQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.530114438 (562h33m34.401197484s) [noise=true , address=45.91.116.7:4010 , piece_id=OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.530439938 (562h33m34.401522973s) [noise=true , address=145.79.227.50:4375 , piece_id=PDUEQK74QRHZDTDIMVJ34PJKA73M4B5XY5XJZSVPZ44Z27HCBODQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.531664770 (562h33m34.402747815s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.532786255 (562h33m34.40386929s) [noise=true , address=145.79.227.37:4566 , piece_id=2XIJUYFKUNRKL65BQHVDF2HHBX5OJW5PFCIKECLBIA7H7B2S4DHQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.533884036 (562h33m34.404967081s) [noise=true , address=145.79.227.13:4804 , piece_id=P35J5NWZZKUQ42RIURXW4EZ6PE5TLCCRU3PXXSOLROSK2NFVLLRA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.535227283 (562h33m34.406310318s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.536530005 (562h33m34.40761304s) [noise=true , address=145.79.227.16:4911 , piece_id=4T2IS6C4ZVXNDJJGMP4AFXHUM6GUI6QIRQTLDQMHWUTFWT77TBPA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.537574183 (562h33m34.408657228s) [noise=true , address=145.79.227.52:4447 , piece_id=EEZTWIBPJZ6H3R2X4X5T2LQMGY2PW5S24NYPGS5SSNBQKPPBH42Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.538175668 (562h33m34.409258703s) [noise=true , address=145.79.227.42:4122 , piece_id=YWMZ4I5FMB6XE5NOI6DTR5PCWBTURCWUOPI273EPNK7UDRAH35YA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.539065993 (562h33m34.410149028s) [noise=true , address=77.90.31.11:4554 , piece_id=XLQUJ327AW6LTSJ6JPT3RV7JJY62226DKPB7TW5YP3WIHLSZA3OA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:36.242152402 (562h33m36.113235437s) [upload=493222]: Next returned nil segment Dec 29 02:02:36.242170537 (562h33m36.113253572s) [upload=493222]: Waiting for error group managing segments... Dec 29 02:02:36.550230588 (562h33m36.421313623s) [noise=true , address=145.79.227.8:4330 , piece_id=RWLDKHK3TEKQJLTBGG3OCBIHQLN74VQXHFB3C5YFKIIPAQDLOOMQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.554730607 (562h33m36.425813662s) [noise=true , address=145.79.227.52:4447 , piece_id=EEZTWIBPJZ6H3R2X4X5T2LQMGY2PW5S24NYPGS5SSNBQKPPBH42Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.559230475 (562h33m36.43031352s) [noise=true , address=145.79.227.37:4566 , piece_id=2XIJUYFKUNRKL65BQHVDF2HHBX5OJW5PFCIKECLBIA7H7B2S4DHQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.565111814 (562h33m36.436194859s) [noise=true , address=145.79.227.50:4375 , piece_id=PDUEQK74QRHZDTDIMVJ34PJKA73M4B5XY5XJZSVPZ44Z27HCBODQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.569171935 (562h33m36.44025497s) [noise=true , address=145.79.227.11:4731 , piece_id=KCDJJQZX6AXJL3DDN662JZIFZ5VOVAQ53EZE7VYA5TABAFZE677Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.587748166 (562h33m36.458831201s) [noise=true , address=145.79.227.13:4804 , piece_id=P35J5NWZZKUQ42RIURXW4EZ6PE5TLCCRU3PXXSOLROSK2NFVLLRA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.599729089 (562h33m36.470812124s) [noise=true , address=145.79.227.42:4122 , piece_id=YWMZ4I5FMB6XE5NOI6DTR5PCWBTURCWUOPI273EPNK7UDRAH35YA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.604479735 (562h33m36.47556278s) [noise=true , address=145.79.227.45:4230 , piece_id=TJYP2EE5H5IYRNYVJ33CWMGGQ5XCYIQBT7ZABBZCEGAU4PKAKB3A, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.717715399 (562h33m36.588798434s) [noise=true , address=145.79.227.16:4911 , piece_id=4T2IS6C4ZVXNDJJGMP4AFXHUM6GUI6QIRQTLDQMHWUTFWT77TBPA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.766466466 (562h33m36.637549501s) [noise=true , address=45.91.116.13:4156 , piece_id=J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.775211488 (562h33m36.646294523s) [noise=true , address=45.91.116.8:4045 , piece_id=NRJYAMG75VAZAK7A3RN4JUW42CHFEHWA72MUUXFHGRXGGY67LCWA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788452180 (562h33m36.659535215s) [noise=true , address=45.91.116.7:4010 , piece_id=OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788651358 (562h33m36.659734394s) [noise=true , address=77.90.31.11:4554 , piece_id=XLQUJ327AW6LTSJ6JPT3RV7JJY62226DKPB7TW5YP3WIHLSZA3OA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788709419 (562h33m36.659792454s) [seg_pos={0 0}, upload=493222]: Segment reached optimal threshold of 13 pieces. Dec 29 02:02:36.788926823 (562h33m36.660009858s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 29 02:02:36.788937053 (562h33m36.660020088s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.788992097 (562h33m36.660075133s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection Dec 29 02:02:36.789000413 (562h33m36.660083448s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.789158274 (562h33m36.660241309s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 02:02:36.789165438 (562h33m36.660248473s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.789184955 (562h33m36.66026799s) [seg_pos={0 0}, upload=493222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:02:36.789276379 (562h33m36.660359414s) [upload=493222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:02:36.789281578 (562h33m36.660364613s) [upload=493222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:02:37.087173919 (562h33m36.958256964s) [upload=493222]: Done uploading segments... Dec 29 02:02:37.087194218 (562h33m36.958277264s) [upload=493222]: Upload finished. err: <nil> Duration: 5.876109413s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X....XXX..XX................XX....X.....X......X...X...] duration: 5.774553269s trace: https://jaeger.dev.storj.io/trace/4d643c0096eb6966 FAIL (2025-12-29 02:04:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA) context deadline exceeded ======LOGS====== 9 02:02:34.282577299 (562h33m34.153660344s) [upload=493222]: Starting upload Dec 29 02:02:34.282654466 (562h33m34.153737501s) [upload=493222]: Done starting upload Dec 29 02:02:34.282739407 (562h33m34.153822452s) [upload=493222]: Uploading segments... Dec 29 02:02:34.282966710 (562h33m34.154049755s) [upload=493222]: Got next segment. Inline: false Dec 29 02:02:34.283001496 (562h33m34.154084532s) [upload=493222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:02:34.283005694 (562h33m34.154088739s) [upload=493222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:02:34.527560786 (562h33m34.398643831s) [seg_pos={0 0}, upload=493222]: Begin upload segment... Dec 29 02:02:34.527626991 (562h33m34.398710036s) [seg_pos={0 0}, upload=493222]: Done begin upload segment. Dec 29 02:02:34.527648643 (562h33m34.398731678s) [noise=true , address=45.91.116.8:4045 , piece_id=NRJYAMG75VAZAK7A3RN4JUW42CHFEHWA72MUUXFHGRXGGY67LCWA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.527721772 (562h33m34.398804817s) [noise=true , address=145.79.227.11:4731 , piece_id=KCDJJQZX6AXJL3DDN662JZIFZ5VOVAQ53EZE7VYA5TABAFZE677Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528050358 (562h33m34.399133393s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528330812 (562h33m34.399413847s) [noise=true , address=45.91.116.13:4156 , piece_id=J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528610094 (562h33m34.399693129s) [noise=true , address=145.79.227.45:4230 , piece_id=TJYP2EE5H5IYRNYVJ33CWMGGQ5XCYIQBT7ZABBZCEGAU4PKAKB3A, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528872794 (562h33m34.399955829s) [noise=true , address=145.79.227.8:4330 , piece_id=RWLDKHK3TEKQJLTBGG3OCBIHQLN74VQXHFB3C5YFKIIPAQDLOOMQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.530114438 (562h33m34.401197484s) [noise=true , address=45.91.116.7:4010 , piece_id=OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.530439938 (562h33m34.401522973s) [noise=true , address=145.79.227.50:4375 , piece_id=PDUEQK74QRHZDTDIMVJ34PJKA73M4B5XY5XJZSVPZ44Z27HCBODQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.531664770 (562h33m34.402747815s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.532786255 (562h33m34.40386929s) [noise=true , address=145.79.227.37:4566 , piece_id=2XIJUYFKUNRKL65BQHVDF2HHBX5OJW5PFCIKECLBIA7H7B2S4DHQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.533884036 (562h33m34.404967081s) [noise=true , address=145.79.227.13:4804 , piece_id=P35J5NWZZKUQ42RIURXW4EZ6PE5TLCCRU3PXXSOLROSK2NFVLLRA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.535227283 (562h33m34.406310318s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.536530005 (562h33m34.40761304s) [noise=true , address=145.79.227.16:4911 , piece_id=4T2IS6C4ZVXNDJJGMP4AFXHUM6GUI6QIRQTLDQMHWUTFWT77TBPA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.537574183 (562h33m34.408657228s) [noise=true , address=145.79.227.52:4447 , piece_id=EEZTWIBPJZ6H3R2X4X5T2LQMGY2PW5S24NYPGS5SSNBQKPPBH42Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.538175668 (562h33m34.409258703s) [noise=true , address=145.79.227.42:4122 , piece_id=YWMZ4I5FMB6XE5NOI6DTR5PCWBTURCWUOPI273EPNK7UDRAH35YA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.539065993 (562h33m34.410149028s) [noise=true , address=77.90.31.11:4554 , piece_id=XLQUJ327AW6LTSJ6JPT3RV7JJY62226DKPB7TW5YP3WIHLSZA3OA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:36.242152402 (562h33m36.113235437s) [upload=493222]: Next returned nil segment Dec 29 02:02:36.242170537 (562h33m36.113253572s) [upload=493222]: Waiting for error group managing segments... Dec 29 02:02:36.550230588 (562h33m36.421313623s) [noise=true , address=145.79.227.8:4330 , piece_id=RWLDKHK3TEKQJLTBGG3OCBIHQLN74VQXHFB3C5YFKIIPAQDLOOMQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.554730607 (562h33m36.425813662s) [noise=true , address=145.79.227.52:4447 , piece_id=EEZTWIBPJZ6H3R2X4X5T2LQMGY2PW5S24NYPGS5SSNBQKPPBH42Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.559230475 (562h33m36.43031352s) [noise=true , address=145.79.227.37:4566 , piece_id=2XIJUYFKUNRKL65BQHVDF2HHBX5OJW5PFCIKECLBIA7H7B2S4DHQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.565111814 (562h33m36.436194859s) [noise=true , address=145.79.227.50:4375 , piece_id=PDUEQK74QRHZDTDIMVJ34PJKA73M4B5XY5XJZSVPZ44Z27HCBODQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.569171935 (562h33m36.44025497s) [noise=true , address=145.79.227.11:4731 , piece_id=KCDJJQZX6AXJL3DDN662JZIFZ5VOVAQ53EZE7VYA5TABAFZE677Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.587748166 (562h33m36.458831201s) [noise=true , address=145.79.227.13:4804 , piece_id=P35J5NWZZKUQ42RIURXW4EZ6PE5TLCCRU3PXXSOLROSK2NFVLLRA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.599729089 (562h33m36.470812124s) [noise=true , address=145.79.227.42:4122 , piece_id=YWMZ4I5FMB6XE5NOI6DTR5PCWBTURCWUOPI273EPNK7UDRAH35YA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.604479735 (562h33m36.47556278s) [noise=true , address=145.79.227.45:4230 , piece_id=TJYP2EE5H5IYRNYVJ33CWMGGQ5XCYIQBT7ZABBZCEGAU4PKAKB3A, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.717715399 (562h33m36.588798434s) [noise=true , address=145.79.227.16:4911 , piece_id=4T2IS6C4ZVXNDJJGMP4AFXHUM6GUI6QIRQTLDQMHWUTFWT77TBPA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.766466466 (562h33m36.637549501s) [noise=true , address=45.91.116.13:4156 , piece_id=J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.775211488 (562h33m36.646294523s) [noise=true , address=45.91.116.8:4045 , piece_id=NRJYAMG75VAZAK7A3RN4JUW42CHFEHWA72MUUXFHGRXGGY67LCWA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788452180 (562h33m36.659535215s) [noise=true , address=45.91.116.7:4010 , piece_id=OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788651358 (562h33m36.659734394s) [noise=true , address=77.90.31.11:4554 , piece_id=XLQUJ327AW6LTSJ6JPT3RV7JJY62226DKPB7TW5YP3WIHLSZA3OA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788709419 (562h33m36.659792454s) [seg_pos={0 0}, upload=493222]: Segment reached optimal threshold of 13 pieces. Dec 29 02:02:36.788926823 (562h33m36.660009858s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 29 02:02:36.788937053 (562h33m36.660020088s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.788992097 (562h33m36.660075133s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection Dec 29 02:02:36.789000413 (562h33m36.660083448s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.789158274 (562h33m36.660241309s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 02:02:36.789165438 (562h33m36.660248473s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.789184955 (562h33m36.66026799s) [seg_pos={0 0}, upload=493222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:02:36.789276379 (562h33m36.660359414s) [upload=493222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:02:36.789281578 (562h33m36.660364613s) [upload=493222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:02:37.087173919 (562h33m36.958256964s) [upload=493222]: Done uploading segments... Dec 29 02:02:37.087194218 (562h33m36.958277264s) [upload=493222]: Upload finished. err: <nil> Duration: 5.774595129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X....XXX..XX................XX....X.....X......X...X..] duration: 5.985673868s trace: https://jaeger.dev.storj.io/trace/4d643c0096eb6966 FAIL (2025-12-29 02:03:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA) context deadline exceeded ======LOGS====== 9 02:02:34.282577299 (562h33m34.153660344s) [upload=493222]: Starting upload Dec 29 02:02:34.282654466 (562h33m34.153737501s) [upload=493222]: Done starting upload Dec 29 02:02:34.282739407 (562h33m34.153822452s) [upload=493222]: Uploading segments... Dec 29 02:02:34.282966710 (562h33m34.154049755s) [upload=493222]: Got next segment. Inline: false Dec 29 02:02:34.283001496 (562h33m34.154084532s) [upload=493222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:02:34.283005694 (562h33m34.154088739s) [upload=493222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:02:34.527560786 (562h33m34.398643831s) [seg_pos={0 0}, upload=493222]: Begin upload segment... Dec 29 02:02:34.527626991 (562h33m34.398710036s) [seg_pos={0 0}, upload=493222]: Done begin upload segment. Dec 29 02:02:34.527648643 (562h33m34.398731678s) [noise=true , address=45.91.116.8:4045 , piece_id=NRJYAMG75VAZAK7A3RN4JUW42CHFEHWA72MUUXFHGRXGGY67LCWA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.527721772 (562h33m34.398804817s) [noise=true , address=145.79.227.11:4731 , piece_id=KCDJJQZX6AXJL3DDN662JZIFZ5VOVAQ53EZE7VYA5TABAFZE677Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528050358 (562h33m34.399133393s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528330812 (562h33m34.399413847s) [noise=true , address=45.91.116.13:4156 , piece_id=J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528610094 (562h33m34.399693129s) [noise=true , address=145.79.227.45:4230 , piece_id=TJYP2EE5H5IYRNYVJ33CWMGGQ5XCYIQBT7ZABBZCEGAU4PKAKB3A, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528872794 (562h33m34.399955829s) [noise=true , address=145.79.227.8:4330 , piece_id=RWLDKHK3TEKQJLTBGG3OCBIHQLN74VQXHFB3C5YFKIIPAQDLOOMQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.530114438 (562h33m34.401197484s) [noise=true , address=45.91.116.7:4010 , piece_id=OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.530439938 (562h33m34.401522973s) [noise=true , address=145.79.227.50:4375 , piece_id=PDUEQK74QRHZDTDIMVJ34PJKA73M4B5XY5XJZSVPZ44Z27HCBODQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.531664770 (562h33m34.402747815s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.532786255 (562h33m34.40386929s) [noise=true , address=145.79.227.37:4566 , piece_id=2XIJUYFKUNRKL65BQHVDF2HHBX5OJW5PFCIKECLBIA7H7B2S4DHQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.533884036 (562h33m34.404967081s) [noise=true , address=145.79.227.13:4804 , piece_id=P35J5NWZZKUQ42RIURXW4EZ6PE5TLCCRU3PXXSOLROSK2NFVLLRA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.535227283 (562h33m34.406310318s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.536530005 (562h33m34.40761304s) [noise=true , address=145.79.227.16:4911 , piece_id=4T2IS6C4ZVXNDJJGMP4AFXHUM6GUI6QIRQTLDQMHWUTFWT77TBPA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.537574183 (562h33m34.408657228s) [noise=true , address=145.79.227.52:4447 , piece_id=EEZTWIBPJZ6H3R2X4X5T2LQMGY2PW5S24NYPGS5SSNBQKPPBH42Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.538175668 (562h33m34.409258703s) [noise=true , address=145.79.227.42:4122 , piece_id=YWMZ4I5FMB6XE5NOI6DTR5PCWBTURCWUOPI273EPNK7UDRAH35YA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.539065993 (562h33m34.410149028s) [noise=true , address=77.90.31.11:4554 , piece_id=XLQUJ327AW6LTSJ6JPT3RV7JJY62226DKPB7TW5YP3WIHLSZA3OA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:36.242152402 (562h33m36.113235437s) [upload=493222]: Next returned nil segment Dec 29 02:02:36.242170537 (562h33m36.113253572s) [upload=493222]: Waiting for error group managing segments... Dec 29 02:02:36.550230588 (562h33m36.421313623s) [noise=true , address=145.79.227.8:4330 , piece_id=RWLDKHK3TEKQJLTBGG3OCBIHQLN74VQXHFB3C5YFKIIPAQDLOOMQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.554730607 (562h33m36.425813662s) [noise=true , address=145.79.227.52:4447 , piece_id=EEZTWIBPJZ6H3R2X4X5T2LQMGY2PW5S24NYPGS5SSNBQKPPBH42Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.559230475 (562h33m36.43031352s) [noise=true , address=145.79.227.37:4566 , piece_id=2XIJUYFKUNRKL65BQHVDF2HHBX5OJW5PFCIKECLBIA7H7B2S4DHQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.565111814 (562h33m36.436194859s) [noise=true , address=145.79.227.50:4375 , piece_id=PDUEQK74QRHZDTDIMVJ34PJKA73M4B5XY5XJZSVPZ44Z27HCBODQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.569171935 (562h33m36.44025497s) [noise=true , address=145.79.227.11:4731 , piece_id=KCDJJQZX6AXJL3DDN662JZIFZ5VOVAQ53EZE7VYA5TABAFZE677Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.587748166 (562h33m36.458831201s) [noise=true , address=145.79.227.13:4804 , piece_id=P35J5NWZZKUQ42RIURXW4EZ6PE5TLCCRU3PXXSOLROSK2NFVLLRA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.599729089 (562h33m36.470812124s) [noise=true , address=145.79.227.42:4122 , piece_id=YWMZ4I5FMB6XE5NOI6DTR5PCWBTURCWUOPI273EPNK7UDRAH35YA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.604479735 (562h33m36.47556278s) [noise=true , address=145.79.227.45:4230 , piece_id=TJYP2EE5H5IYRNYVJ33CWMGGQ5XCYIQBT7ZABBZCEGAU4PKAKB3A, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.717715399 (562h33m36.588798434s) [noise=true , address=145.79.227.16:4911 , piece_id=4T2IS6C4ZVXNDJJGMP4AFXHUM6GUI6QIRQTLDQMHWUTFWT77TBPA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.766466466 (562h33m36.637549501s) [noise=true , address=45.91.116.13:4156 , piece_id=J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.775211488 (562h33m36.646294523s) [noise=true , address=45.91.116.8:4045 , piece_id=NRJYAMG75VAZAK7A3RN4JUW42CHFEHWA72MUUXFHGRXGGY67LCWA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788452180 (562h33m36.659535215s) [noise=true , address=45.91.116.7:4010 , piece_id=OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788651358 (562h33m36.659734394s) [noise=true , address=77.90.31.11:4554 , piece_id=XLQUJ327AW6LTSJ6JPT3RV7JJY62226DKPB7TW5YP3WIHLSZA3OA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788709419 (562h33m36.659792454s) [seg_pos={0 0}, upload=493222]: Segment reached optimal threshold of 13 pieces. Dec 29 02:02:36.788926823 (562h33m36.660009858s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 29 02:02:36.788937053 (562h33m36.660020088s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.788992097 (562h33m36.660075133s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection Dec 29 02:02:36.789000413 (562h33m36.660083448s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.789158274 (562h33m36.660241309s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 02:02:36.789165438 (562h33m36.660248473s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.789184955 (562h33m36.66026799s) [seg_pos={0 0}, upload=493222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:02:36.789276379 (562h33m36.660359414s) [upload=493222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:02:36.789281578 (562h33m36.660364613s) [upload=493222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:02:37.087173919 (562h33m36.958256964s) [upload=493222]: Done uploading segments... Dec 29 02:02:37.087194218 (562h33m36.958277264s) [upload=493222]: Upload finished. err: <nil> Duration: 5.985696212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X......X....XXX..XX................XX....X.....X......X...X.] duration: 5.890106256s trace: https://jaeger.dev.storj.io/trace/4d643c0096eb6966 FAIL (2025-12-29 02:03:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA) context deadline exceeded ======LOGS====== 9 02:02:34.282577299 (562h33m34.153660344s) [upload=493222]: Starting upload Dec 29 02:02:34.282654466 (562h33m34.153737501s) [upload=493222]: Done starting upload Dec 29 02:02:34.282739407 (562h33m34.153822452s) [upload=493222]: Uploading segments... Dec 29 02:02:34.282966710 (562h33m34.154049755s) [upload=493222]: Got next segment. Inline: false Dec 29 02:02:34.283001496 (562h33m34.154084532s) [upload=493222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:02:34.283005694 (562h33m34.154088739s) [upload=493222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:02:34.527560786 (562h33m34.398643831s) [seg_pos={0 0}, upload=493222]: Begin upload segment... Dec 29 02:02:34.527626991 (562h33m34.398710036s) [seg_pos={0 0}, upload=493222]: Done begin upload segment. Dec 29 02:02:34.527648643 (562h33m34.398731678s) [noise=true , address=45.91.116.8:4045 , piece_id=NRJYAMG75VAZAK7A3RN4JUW42CHFEHWA72MUUXFHGRXGGY67LCWA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.527721772 (562h33m34.398804817s) [noise=true , address=145.79.227.11:4731 , piece_id=KCDJJQZX6AXJL3DDN662JZIFZ5VOVAQ53EZE7VYA5TABAFZE677Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528050358 (562h33m34.399133393s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528330812 (562h33m34.399413847s) [noise=true , address=45.91.116.13:4156 , piece_id=J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528610094 (562h33m34.399693129s) [noise=true , address=145.79.227.45:4230 , piece_id=TJYP2EE5H5IYRNYVJ33CWMGGQ5XCYIQBT7ZABBZCEGAU4PKAKB3A, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528872794 (562h33m34.399955829s) [noise=true , address=145.79.227.8:4330 , piece_id=RWLDKHK3TEKQJLTBGG3OCBIHQLN74VQXHFB3C5YFKIIPAQDLOOMQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.530114438 (562h33m34.401197484s) [noise=true , address=45.91.116.7:4010 , piece_id=OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.530439938 (562h33m34.401522973s) [noise=true , address=145.79.227.50:4375 , piece_id=PDUEQK74QRHZDTDIMVJ34PJKA73M4B5XY5XJZSVPZ44Z27HCBODQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.531664770 (562h33m34.402747815s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.532786255 (562h33m34.40386929s) [noise=true , address=145.79.227.37:4566 , piece_id=2XIJUYFKUNRKL65BQHVDF2HHBX5OJW5PFCIKECLBIA7H7B2S4DHQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.533884036 (562h33m34.404967081s) [noise=true , address=145.79.227.13:4804 , piece_id=P35J5NWZZKUQ42RIURXW4EZ6PE5TLCCRU3PXXSOLROSK2NFVLLRA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.535227283 (562h33m34.406310318s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.536530005 (562h33m34.40761304s) [noise=true , address=145.79.227.16:4911 , piece_id=4T2IS6C4ZVXNDJJGMP4AFXHUM6GUI6QIRQTLDQMHWUTFWT77TBPA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.537574183 (562h33m34.408657228s) [noise=true , address=145.79.227.52:4447 , piece_id=EEZTWIBPJZ6H3R2X4X5T2LQMGY2PW5S24NYPGS5SSNBQKPPBH42Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.538175668 (562h33m34.409258703s) [noise=true , address=145.79.227.42:4122 , piece_id=YWMZ4I5FMB6XE5NOI6DTR5PCWBTURCWUOPI273EPNK7UDRAH35YA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.539065993 (562h33m34.410149028s) [noise=true , address=77.90.31.11:4554 , piece_id=XLQUJ327AW6LTSJ6JPT3RV7JJY62226DKPB7TW5YP3WIHLSZA3OA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:36.242152402 (562h33m36.113235437s) [upload=493222]: Next returned nil segment Dec 29 02:02:36.242170537 (562h33m36.113253572s) [upload=493222]: Waiting for error group managing segments... Dec 29 02:02:36.550230588 (562h33m36.421313623s) [noise=true , address=145.79.227.8:4330 , piece_id=RWLDKHK3TEKQJLTBGG3OCBIHQLN74VQXHFB3C5YFKIIPAQDLOOMQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.554730607 (562h33m36.425813662s) [noise=true , address=145.79.227.52:4447 , piece_id=EEZTWIBPJZ6H3R2X4X5T2LQMGY2PW5S24NYPGS5SSNBQKPPBH42Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.559230475 (562h33m36.43031352s) [noise=true , address=145.79.227.37:4566 , piece_id=2XIJUYFKUNRKL65BQHVDF2HHBX5OJW5PFCIKECLBIA7H7B2S4DHQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.565111814 (562h33m36.436194859s) [noise=true , address=145.79.227.50:4375 , piece_id=PDUEQK74QRHZDTDIMVJ34PJKA73M4B5XY5XJZSVPZ44Z27HCBODQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.569171935 (562h33m36.44025497s) [noise=true , address=145.79.227.11:4731 , piece_id=KCDJJQZX6AXJL3DDN662JZIFZ5VOVAQ53EZE7VYA5TABAFZE677Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.587748166 (562h33m36.458831201s) [noise=true , address=145.79.227.13:4804 , piece_id=P35J5NWZZKUQ42RIURXW4EZ6PE5TLCCRU3PXXSOLROSK2NFVLLRA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.599729089 (562h33m36.470812124s) [noise=true , address=145.79.227.42:4122 , piece_id=YWMZ4I5FMB6XE5NOI6DTR5PCWBTURCWUOPI273EPNK7UDRAH35YA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.604479735 (562h33m36.47556278s) [noise=true , address=145.79.227.45:4230 , piece_id=TJYP2EE5H5IYRNYVJ33CWMGGQ5XCYIQBT7ZABBZCEGAU4PKAKB3A, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.717715399 (562h33m36.588798434s) [noise=true , address=145.79.227.16:4911 , piece_id=4T2IS6C4ZVXNDJJGMP4AFXHUM6GUI6QIRQTLDQMHWUTFWT77TBPA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.766466466 (562h33m36.637549501s) [noise=true , address=45.91.116.13:4156 , piece_id=J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.775211488 (562h33m36.646294523s) [noise=true , address=45.91.116.8:4045 , piece_id=NRJYAMG75VAZAK7A3RN4JUW42CHFEHWA72MUUXFHGRXGGY67LCWA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788452180 (562h33m36.659535215s) [noise=true , address=45.91.116.7:4010 , piece_id=OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788651358 (562h33m36.659734394s) [noise=true , address=77.90.31.11:4554 , piece_id=XLQUJ327AW6LTSJ6JPT3RV7JJY62226DKPB7TW5YP3WIHLSZA3OA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788709419 (562h33m36.659792454s) [seg_pos={0 0}, upload=493222]: Segment reached optimal threshold of 13 pieces. Dec 29 02:02:36.788926823 (562h33m36.660009858s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 29 02:02:36.788937053 (562h33m36.660020088s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.788992097 (562h33m36.660075133s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection Dec 29 02:02:36.789000413 (562h33m36.660083448s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.789158274 (562h33m36.660241309s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 02:02:36.789165438 (562h33m36.660248473s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.789184955 (562h33m36.66026799s) [seg_pos={0 0}, upload=493222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:02:36.789276379 (562h33m36.660359414s) [upload=493222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:02:36.789281578 (562h33m36.660364613s) [upload=493222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:02:37.087173919 (562h33m36.958256964s) [upload=493222]: Done uploading segments... Dec 29 02:02:37.087194218 (562h33m36.958277264s) [upload=493222]: Upload finished. err: <nil> Duration: 5.890126105s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X......X....XXX..XX................XX....X.....X......X...X] duration: 30.003134122s trace: https://jaeger.dev.storj.io/trace/4d643c0096eb6966 FAIL (2025-12-29 02:03:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA) context deadline exceeded ======LOGS====== 9 02:02:34.282577299 (562h33m34.153660344s) [upload=493222]: Starting upload Dec 29 02:02:34.282654466 (562h33m34.153737501s) [upload=493222]: Done starting upload Dec 29 02:02:34.282739407 (562h33m34.153822452s) [upload=493222]: Uploading segments... Dec 29 02:02:34.282966710 (562h33m34.154049755s) [upload=493222]: Got next segment. Inline: false Dec 29 02:02:34.283001496 (562h33m34.154084532s) [upload=493222]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 02:02:34.283005694 (562h33m34.154088739s) [upload=493222]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 02:02:34.527560786 (562h33m34.398643831s) [seg_pos={0 0}, upload=493222]: Begin upload segment... Dec 29 02:02:34.527626991 (562h33m34.398710036s) [seg_pos={0 0}, upload=493222]: Done begin upload segment. Dec 29 02:02:34.527648643 (562h33m34.398731678s) [noise=true , address=45.91.116.8:4045 , piece_id=NRJYAMG75VAZAK7A3RN4JUW42CHFEHWA72MUUXFHGRXGGY67LCWA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.527721772 (562h33m34.398804817s) [noise=true , address=145.79.227.11:4731 , piece_id=KCDJJQZX6AXJL3DDN662JZIFZ5VOVAQ53EZE7VYA5TABAFZE677Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528050358 (562h33m34.399133393s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528330812 (562h33m34.399413847s) [noise=true , address=45.91.116.13:4156 , piece_id=J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528610094 (562h33m34.399693129s) [noise=true , address=145.79.227.45:4230 , piece_id=TJYP2EE5H5IYRNYVJ33CWMGGQ5XCYIQBT7ZABBZCEGAU4PKAKB3A, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.528872794 (562h33m34.399955829s) [noise=true , address=145.79.227.8:4330 , piece_id=RWLDKHK3TEKQJLTBGG3OCBIHQLN74VQXHFB3C5YFKIIPAQDLOOMQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.530114438 (562h33m34.401197484s) [noise=true , address=45.91.116.7:4010 , piece_id=OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.530439938 (562h33m34.401522973s) [noise=true , address=145.79.227.50:4375 , piece_id=PDUEQK74QRHZDTDIMVJ34PJKA73M4B5XY5XJZSVPZ44Z27HCBODQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.531664770 (562h33m34.402747815s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.532786255 (562h33m34.40386929s) [noise=true , address=145.79.227.37:4566 , piece_id=2XIJUYFKUNRKL65BQHVDF2HHBX5OJW5PFCIKECLBIA7H7B2S4DHQ, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.533884036 (562h33m34.404967081s) [noise=true , address=145.79.227.13:4804 , piece_id=P35J5NWZZKUQ42RIURXW4EZ6PE5TLCCRU3PXXSOLROSK2NFVLLRA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.535227283 (562h33m34.406310318s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.536530005 (562h33m34.40761304s) [noise=true , address=145.79.227.16:4911 , piece_id=4T2IS6C4ZVXNDJJGMP4AFXHUM6GUI6QIRQTLDQMHWUTFWT77TBPA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.537574183 (562h33m34.408657228s) [noise=true , address=145.79.227.52:4447 , piece_id=EEZTWIBPJZ6H3R2X4X5T2LQMGY2PW5S24NYPGS5SSNBQKPPBH42Q, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.538175668 (562h33m34.409258703s) [noise=true , address=145.79.227.42:4122 , piece_id=YWMZ4I5FMB6XE5NOI6DTR5PCWBTURCWUOPI273EPNK7UDRAH35YA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:34.539065993 (562h33m34.410149028s) [noise=true , address=77.90.31.11:4554 , piece_id=XLQUJ327AW6LTSJ6JPT3RV7JJY62226DKPB7TW5YP3WIHLSZA3OA, seg_pos={0 0}, upload=493222]: Uploading piece... Dec 29 02:02:36.242152402 (562h33m36.113235437s) [upload=493222]: Next returned nil segment Dec 29 02:02:36.242170537 (562h33m36.113253572s) [upload=493222]: Waiting for error group managing segments... Dec 29 02:02:36.550230588 (562h33m36.421313623s) [noise=true , address=145.79.227.8:4330 , piece_id=RWLDKHK3TEKQJLTBGG3OCBIHQLN74VQXHFB3C5YFKIIPAQDLOOMQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.554730607 (562h33m36.425813662s) [noise=true , address=145.79.227.52:4447 , piece_id=EEZTWIBPJZ6H3R2X4X5T2LQMGY2PW5S24NYPGS5SSNBQKPPBH42Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.559230475 (562h33m36.43031352s) [noise=true , address=145.79.227.37:4566 , piece_id=2XIJUYFKUNRKL65BQHVDF2HHBX5OJW5PFCIKECLBIA7H7B2S4DHQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.565111814 (562h33m36.436194859s) [noise=true , address=145.79.227.50:4375 , piece_id=PDUEQK74QRHZDTDIMVJ34PJKA73M4B5XY5XJZSVPZ44Z27HCBODQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.569171935 (562h33m36.44025497s) [noise=true , address=145.79.227.11:4731 , piece_id=KCDJJQZX6AXJL3DDN662JZIFZ5VOVAQ53EZE7VYA5TABAFZE677Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.587748166 (562h33m36.458831201s) [noise=true , address=145.79.227.13:4804 , piece_id=P35J5NWZZKUQ42RIURXW4EZ6PE5TLCCRU3PXXSOLROSK2NFVLLRA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.599729089 (562h33m36.470812124s) [noise=true , address=145.79.227.42:4122 , piece_id=YWMZ4I5FMB6XE5NOI6DTR5PCWBTURCWUOPI273EPNK7UDRAH35YA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.604479735 (562h33m36.47556278s) [noise=true , address=145.79.227.45:4230 , piece_id=TJYP2EE5H5IYRNYVJ33CWMGGQ5XCYIQBT7ZABBZCEGAU4PKAKB3A, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.717715399 (562h33m36.588798434s) [noise=true , address=145.79.227.16:4911 , piece_id=4T2IS6C4ZVXNDJJGMP4AFXHUM6GUI6QIRQTLDQMHWUTFWT77TBPA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.766466466 (562h33m36.637549501s) [noise=true , address=45.91.116.13:4156 , piece_id=J2C4PNXEK2N6RDQ7A7CLMQ4J4E72N6UGXXZLSYJWGNZKKYXU5JIA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.775211488 (562h33m36.646294523s) [noise=true , address=45.91.116.8:4045 , piece_id=NRJYAMG75VAZAK7A3RN4JUW42CHFEHWA72MUUXFHGRXGGY67LCWA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788452180 (562h33m36.659535215s) [noise=true , address=45.91.116.7:4010 , piece_id=OBLMHP4NEIWBXWEYABFLTX3XWM2XSFE4Z5FGSXQOVT2VCIWPWJXA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788651358 (562h33m36.659734394s) [noise=true , address=77.90.31.11:4554 , piece_id=XLQUJ327AW6LTSJ6JPT3RV7JJY62226DKPB7TW5YP3WIHLSZA3OA, seg_pos={0 0}, upload=493222]: Done uploading piece. err: <nil> Dec 29 02:02:36.788709419 (562h33m36.659792454s) [seg_pos={0 0}, upload=493222]: Segment reached optimal threshold of 13 pieces. Dec 29 02:02:36.788926823 (562h33m36.660009858s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 29 02:02:36.788937053 (562h33m36.660020088s) [noise=true , address=45.91.116.5:4300 , piece_id=PSJW6RYEUMVU4CINSOHIG4ZCVXP7NZ3MDHPVDWTVL2U2DUR3RZ5A, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.788992097 (562h33m36.660075133s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57336->145.79.227.2:4113: use of closed network connection Dec 29 02:02:36.789000413 (562h33m36.660083448s) [noise=true , address=145.79.227.2:4113 , piece_id=LVVWDQBI5TDQVG767SPESVRGXRQDX4GU3OMMAIH2M4C362L3RE2Q, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.789158274 (562h33m36.660241309s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 29 02:02:36.789165438 (562h33m36.660248473s) [noise=true , address=145.79.227.34:4906 , piece_id=KDKRLLBE3VDHIELQVGE2KMK37KTEOEPICS6IRMOPTUC42QJ4W5YQ, seg_pos={0 0}, upload=493222]: Long tail cancellation due to optimal threshold Dec 29 02:02:36.789184955 (562h33m36.66026799s) [seg_pos={0 0}, upload=493222]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 02:02:36.789276379 (562h33m36.660359414s) [upload=493222]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:02:36.789281578 (562h33m36.660364613s) [upload=493222]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:02:37.087173919 (562h33m36.958256964s) [upload=493222]: Done uploading segments... Dec 29 02:02:37.087194218 (562h33m36.958277264s) [upload=493222]: Upload finished. err: <nil> Duration: 30.003154722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X......X....XXX..XX................XX....X.....X......X...] duration: 6.327885585s trace: https://jaeger.dev.storj.io/trace/3e1f3039711db9f6 FAIL (2025-12-29 02:02:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 01:59:38.975293575 (562h30m38.84637661s) [upload=493180]: Starting upload Dec 29 01:59:38.975364220 (562h30m38.846447265s) [upload=493180]: Done starting upload Dec 29 01:59:38.975423523 (562h30m38.846506558s) [upload=493180]: Uploading segments... Dec 29 01:59:38.975540005 (562h30m38.84662305s) [upload=493180]: Got next segment. Inline: false Dec 29 01:59:38.975565974 (562h30m38.846649019s) [upload=493180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:59:38.975570132 (562h30m38.846653167s) [upload=493180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:59:39.219880405 (562h30m39.09096344s) [seg_pos={0 0}, upload=493180]: Begin upload segment... Dec 29 01:59:39.219923637 (562h30m39.091006672s) [seg_pos={0 0}, upload=493180]: Done begin upload segment. Dec 29 01:59:39.219955067 (562h30m39.091038102s) [noise=true , address=145.79.227.45:4231 , piece_id=22NRWH7WQKLRHFUZ42FHO7O7SR444WPD4EP3Y3KR6ZYL6RKDCL4A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.219958904 (562h30m39.091041939s) [noise=true , address=145.79.227.48:4301 , piece_id=5OEILVM4RJVX6IZSTNOAAJV3JWLUA7Z3TP6IP6TGUOSPGMMNCAJQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.220464056 (562h30m39.091547091s) [noise=true , address=145.79.227.3:4150 , piece_id=ZJWIISMYPWPAIPV663EFAHEXO2W3FA67I36UHY4KZX326FX5CO6A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221310478 (562h30m39.092393513s) [noise=true , address=145.79.227.13:4802 , piece_id=QVC2GCHERYR55PIOWF43TQ34O3QFN5OWECBUBMUOJOUL36YCDIFA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221650125 (562h30m39.09273317s) [noise=true , address=45.91.116.14:4192 , piece_id=SLUTIZHU5ISKYGYROBUQBFDLCAGPV6XDXQTXHF5XDCCHOP46RQUQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221839355 (562h30m39.09292239s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.222138304 (562h30m39.093221339s) [noise=true , address=45.91.116.8:4048 , piece_id=3T6UZYUX7WEBWZCBNGE7OHYLRPIVH6YJPDBLMMRFTHKFUUQVUM5Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.222428887 (562h30m39.093511922s) [noise=true , address=145.79.227.50:4374 , piece_id=GVMZW5ZMBBPIVAYH5B3HUXWTLRAWIWTXSVMKZJHGI6UEQLWJRGRA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.227274724 (562h30m39.098357759s) [noise=true , address=145.79.227.11:4732 , piece_id=F5RNFOMZ4RYHFRLCO5LM4YLNOMA3SN3PM46RI7RSQCRF7J66WY2Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.229287257 (562h30m39.100370292s) [noise=true , address=145.79.227.2:4113 , piece_id=33NVLK2NBHERFZEFB47KOY5EPWKEB2MQ5WU6ENO6BE3H527O4ADA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.230594666 (562h30m39.101677701s) [noise=true , address=145.79.227.15:4876 , piece_id=7ZD3LEVP3DZEQZC7GLWU6BOPVZPAR542UURXGTXTBCDCMVBIRGLA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.231517844 (562h30m39.102600879s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.232631264 (562h30m39.103714299s) [noise=true , address=145.79.227.47:4265 , piece_id=KK5I5PQN6R5YEQTUQNDSERNV5UKRZ7URBSNCXYI5SLBSEJZRXGAQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.233220306 (562h30m39.104303341s) [noise=true , address=145.79.227.42:4123 , piece_id=KDB7MUWZXIS3QGGAPXLHWSD5ZM7WUSFRGV4AC5DMF5JWODAQ5VHA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.234069172 (562h30m39.105152217s) [noise=true , address=145.79.227.7:4293 , piece_id=IMTTUY5GYUTU5BZAZKXBXWLH3IFFZI3Z6TSUATJ3VLQHSDQJHI3Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.234955439 (562h30m39.106038504s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:40.937306407 (562h30m40.808389442s) [upload=493180]: Next returned nil segment Dec 29 01:59:40.937321756 (562h30m40.808404801s) [upload=493180]: Waiting for error group managing segments... Dec 29 01:59:41.217294798 (562h30m41.088377843s) [noise=true , address=145.79.227.15:4876 , piece_id=7ZD3LEVP3DZEQZC7GLWU6BOPVZPAR542UURXGTXTBCDCMVBIRGLA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.223671520 (562h30m41.094754555s) [noise=true , address=145.79.227.48:4301 , piece_id=5OEILVM4RJVX6IZSTNOAAJV3JWLUA7Z3TP6IP6TGUOSPGMMNCAJQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.227339554 (562h30m41.098422589s) [noise=true , address=145.79.227.2:4113 , piece_id=33NVLK2NBHERFZEFB47KOY5EPWKEB2MQ5WU6ENO6BE3H527O4ADA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.232457159 (562h30m41.103540204s) [noise=true , address=145.79.227.45:4231 , piece_id=22NRWH7WQKLRHFUZ42FHO7O7SR444WPD4EP3Y3KR6ZYL6RKDCL4A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.239167085 (562h30m41.11025013s) [noise=true , address=145.79.227.7:4293 , piece_id=IMTTUY5GYUTU5BZAZKXBXWLH3IFFZI3Z6TSUATJ3VLQHSDQJHI3Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.240513589 (562h30m41.111596634s) [noise=true , address=145.79.227.50:4374 , piece_id=GVMZW5ZMBBPIVAYH5B3HUXWTLRAWIWTXSVMKZJHGI6UEQLWJRGRA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.242912126 (562h30m41.113995172s) [noise=true , address=145.79.227.3:4150 , piece_id=ZJWIISMYPWPAIPV663EFAHEXO2W3FA67I36UHY4KZX326FX5CO6A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.247753995 (562h30m41.11883704s) [noise=true , address=145.79.227.42:4123 , piece_id=KDB7MUWZXIS3QGGAPXLHWSD5ZM7WUSFRGV4AC5DMF5JWODAQ5VHA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.252074161 (562h30m41.123157206s) [noise=true , address=145.79.227.13:4802 , piece_id=QVC2GCHERYR55PIOWF43TQ34O3QFN5OWECBUBMUOJOUL36YCDIFA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.253844002 (562h30m41.124927037s) [noise=true , address=145.79.227.11:4732 , piece_id=F5RNFOMZ4RYHFRLCO5LM4YLNOMA3SN3PM46RI7RSQCRF7J66WY2Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.271486786 (562h30m41.142569821s) [noise=true , address=145.79.227.47:4265 , piece_id=KK5I5PQN6R5YEQTUQNDSERNV5UKRZ7URBSNCXYI5SLBSEJZRXGAQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.454186376 (562h30m41.325269431s) [noise=true , address=45.91.116.8:4048 , piece_id=3T6UZYUX7WEBWZCBNGE7OHYLRPIVH6YJPDBLMMRFTHKFUUQVUM5Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.510719051 (562h30m41.381802096s) [noise=true , address=45.91.116.14:4192 , piece_id=SLUTIZHU5ISKYGYROBUQBFDLCAGPV6XDXQTXHF5XDCCHOP46RQUQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.510746734 (562h30m41.381829769s) [seg_pos={0 0}, upload=493180]: Segment reached optimal threshold of 13 pieces. Dec 29 01:59:41.511169308 (562h30m41.382252343s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511180940 (562h30m41.382263985s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511200658 (562h30m41.382283693s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511217840 (562h30m41.382300875s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511406069 (562h30m41.382489104s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511412521 (562h30m41.382495556s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511431707 (562h30m41.382514742s) [seg_pos={0 0}, upload=493180]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:59:41.511512171 (562h30m41.382595206s) [upload=493180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:59:41.511517241 (562h30m41.382600276s) [upload=493180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:59:41.801536229 (562h30m41.672619274s) [upload=493180]: Done uploading segments... Dec 29 01:59:41.801554483 (562h30m41.672637518s) [upload=493180]: Upload finished. err: <nil> Duration: 6.327916745s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X......X....XXX..XX................XX....X.....X......X..] duration: 5.685349996s trace: https://jaeger.dev.storj.io/trace/3e1f3039711db9f6 FAIL (2025-12-29 02:01:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 01:59:38.975293575 (562h30m38.84637661s) [upload=493180]: Starting upload Dec 29 01:59:38.975364220 (562h30m38.846447265s) [upload=493180]: Done starting upload Dec 29 01:59:38.975423523 (562h30m38.846506558s) [upload=493180]: Uploading segments... Dec 29 01:59:38.975540005 (562h30m38.84662305s) [upload=493180]: Got next segment. Inline: false Dec 29 01:59:38.975565974 (562h30m38.846649019s) [upload=493180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:59:38.975570132 (562h30m38.846653167s) [upload=493180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:59:39.219880405 (562h30m39.09096344s) [seg_pos={0 0}, upload=493180]: Begin upload segment... Dec 29 01:59:39.219923637 (562h30m39.091006672s) [seg_pos={0 0}, upload=493180]: Done begin upload segment. Dec 29 01:59:39.219955067 (562h30m39.091038102s) [noise=true , address=145.79.227.45:4231 , piece_id=22NRWH7WQKLRHFUZ42FHO7O7SR444WPD4EP3Y3KR6ZYL6RKDCL4A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.219958904 (562h30m39.091041939s) [noise=true , address=145.79.227.48:4301 , piece_id=5OEILVM4RJVX6IZSTNOAAJV3JWLUA7Z3TP6IP6TGUOSPGMMNCAJQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.220464056 (562h30m39.091547091s) [noise=true , address=145.79.227.3:4150 , piece_id=ZJWIISMYPWPAIPV663EFAHEXO2W3FA67I36UHY4KZX326FX5CO6A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221310478 (562h30m39.092393513s) [noise=true , address=145.79.227.13:4802 , piece_id=QVC2GCHERYR55PIOWF43TQ34O3QFN5OWECBUBMUOJOUL36YCDIFA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221650125 (562h30m39.09273317s) [noise=true , address=45.91.116.14:4192 , piece_id=SLUTIZHU5ISKYGYROBUQBFDLCAGPV6XDXQTXHF5XDCCHOP46RQUQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221839355 (562h30m39.09292239s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.222138304 (562h30m39.093221339s) [noise=true , address=45.91.116.8:4048 , piece_id=3T6UZYUX7WEBWZCBNGE7OHYLRPIVH6YJPDBLMMRFTHKFUUQVUM5Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.222428887 (562h30m39.093511922s) [noise=true , address=145.79.227.50:4374 , piece_id=GVMZW5ZMBBPIVAYH5B3HUXWTLRAWIWTXSVMKZJHGI6UEQLWJRGRA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.227274724 (562h30m39.098357759s) [noise=true , address=145.79.227.11:4732 , piece_id=F5RNFOMZ4RYHFRLCO5LM4YLNOMA3SN3PM46RI7RSQCRF7J66WY2Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.229287257 (562h30m39.100370292s) [noise=true , address=145.79.227.2:4113 , piece_id=33NVLK2NBHERFZEFB47KOY5EPWKEB2MQ5WU6ENO6BE3H527O4ADA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.230594666 (562h30m39.101677701s) [noise=true , address=145.79.227.15:4876 , piece_id=7ZD3LEVP3DZEQZC7GLWU6BOPVZPAR542UURXGTXTBCDCMVBIRGLA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.231517844 (562h30m39.102600879s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.232631264 (562h30m39.103714299s) [noise=true , address=145.79.227.47:4265 , piece_id=KK5I5PQN6R5YEQTUQNDSERNV5UKRZ7URBSNCXYI5SLBSEJZRXGAQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.233220306 (562h30m39.104303341s) [noise=true , address=145.79.227.42:4123 , piece_id=KDB7MUWZXIS3QGGAPXLHWSD5ZM7WUSFRGV4AC5DMF5JWODAQ5VHA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.234069172 (562h30m39.105152217s) [noise=true , address=145.79.227.7:4293 , piece_id=IMTTUY5GYUTU5BZAZKXBXWLH3IFFZI3Z6TSUATJ3VLQHSDQJHI3Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.234955439 (562h30m39.106038504s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:40.937306407 (562h30m40.808389442s) [upload=493180]: Next returned nil segment Dec 29 01:59:40.937321756 (562h30m40.808404801s) [upload=493180]: Waiting for error group managing segments... Dec 29 01:59:41.217294798 (562h30m41.088377843s) [noise=true , address=145.79.227.15:4876 , piece_id=7ZD3LEVP3DZEQZC7GLWU6BOPVZPAR542UURXGTXTBCDCMVBIRGLA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.223671520 (562h30m41.094754555s) [noise=true , address=145.79.227.48:4301 , piece_id=5OEILVM4RJVX6IZSTNOAAJV3JWLUA7Z3TP6IP6TGUOSPGMMNCAJQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.227339554 (562h30m41.098422589s) [noise=true , address=145.79.227.2:4113 , piece_id=33NVLK2NBHERFZEFB47KOY5EPWKEB2MQ5WU6ENO6BE3H527O4ADA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.232457159 (562h30m41.103540204s) [noise=true , address=145.79.227.45:4231 , piece_id=22NRWH7WQKLRHFUZ42FHO7O7SR444WPD4EP3Y3KR6ZYL6RKDCL4A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.239167085 (562h30m41.11025013s) [noise=true , address=145.79.227.7:4293 , piece_id=IMTTUY5GYUTU5BZAZKXBXWLH3IFFZI3Z6TSUATJ3VLQHSDQJHI3Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.240513589 (562h30m41.111596634s) [noise=true , address=145.79.227.50:4374 , piece_id=GVMZW5ZMBBPIVAYH5B3HUXWTLRAWIWTXSVMKZJHGI6UEQLWJRGRA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.242912126 (562h30m41.113995172s) [noise=true , address=145.79.227.3:4150 , piece_id=ZJWIISMYPWPAIPV663EFAHEXO2W3FA67I36UHY4KZX326FX5CO6A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.247753995 (562h30m41.11883704s) [noise=true , address=145.79.227.42:4123 , piece_id=KDB7MUWZXIS3QGGAPXLHWSD5ZM7WUSFRGV4AC5DMF5JWODAQ5VHA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.252074161 (562h30m41.123157206s) [noise=true , address=145.79.227.13:4802 , piece_id=QVC2GCHERYR55PIOWF43TQ34O3QFN5OWECBUBMUOJOUL36YCDIFA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.253844002 (562h30m41.124927037s) [noise=true , address=145.79.227.11:4732 , piece_id=F5RNFOMZ4RYHFRLCO5LM4YLNOMA3SN3PM46RI7RSQCRF7J66WY2Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.271486786 (562h30m41.142569821s) [noise=true , address=145.79.227.47:4265 , piece_id=KK5I5PQN6R5YEQTUQNDSERNV5UKRZ7URBSNCXYI5SLBSEJZRXGAQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.454186376 (562h30m41.325269431s) [noise=true , address=45.91.116.8:4048 , piece_id=3T6UZYUX7WEBWZCBNGE7OHYLRPIVH6YJPDBLMMRFTHKFUUQVUM5Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.510719051 (562h30m41.381802096s) [noise=true , address=45.91.116.14:4192 , piece_id=SLUTIZHU5ISKYGYROBUQBFDLCAGPV6XDXQTXHF5XDCCHOP46RQUQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.510746734 (562h30m41.381829769s) [seg_pos={0 0}, upload=493180]: Segment reached optimal threshold of 13 pieces. Dec 29 01:59:41.511169308 (562h30m41.382252343s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511180940 (562h30m41.382263985s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511200658 (562h30m41.382283693s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511217840 (562h30m41.382300875s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511406069 (562h30m41.382489104s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511412521 (562h30m41.382495556s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511431707 (562h30m41.382514742s) [seg_pos={0 0}, upload=493180]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:59:41.511512171 (562h30m41.382595206s) [upload=493180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:59:41.511517241 (562h30m41.382600276s) [upload=493180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:59:41.801536229 (562h30m41.672619274s) [upload=493180]: Done uploading segments... Dec 29 01:59:41.801554483 (562h30m41.672637518s) [upload=493180]: Upload finished. err: <nil> Duration: 5.685390283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X......X....XXX..XX................XX....X.....X......X.] duration: 6.462928606s trace: https://jaeger.dev.storj.io/trace/3e1f3039711db9f6 FAIL (2025-12-29 02:00:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 01:59:38.975293575 (562h30m38.84637661s) [upload=493180]: Starting upload Dec 29 01:59:38.975364220 (562h30m38.846447265s) [upload=493180]: Done starting upload Dec 29 01:59:38.975423523 (562h30m38.846506558s) [upload=493180]: Uploading segments... Dec 29 01:59:38.975540005 (562h30m38.84662305s) [upload=493180]: Got next segment. Inline: false Dec 29 01:59:38.975565974 (562h30m38.846649019s) [upload=493180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:59:38.975570132 (562h30m38.846653167s) [upload=493180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:59:39.219880405 (562h30m39.09096344s) [seg_pos={0 0}, upload=493180]: Begin upload segment... Dec 29 01:59:39.219923637 (562h30m39.091006672s) [seg_pos={0 0}, upload=493180]: Done begin upload segment. Dec 29 01:59:39.219955067 (562h30m39.091038102s) [noise=true , address=145.79.227.45:4231 , piece_id=22NRWH7WQKLRHFUZ42FHO7O7SR444WPD4EP3Y3KR6ZYL6RKDCL4A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.219958904 (562h30m39.091041939s) [noise=true , address=145.79.227.48:4301 , piece_id=5OEILVM4RJVX6IZSTNOAAJV3JWLUA7Z3TP6IP6TGUOSPGMMNCAJQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.220464056 (562h30m39.091547091s) [noise=true , address=145.79.227.3:4150 , piece_id=ZJWIISMYPWPAIPV663EFAHEXO2W3FA67I36UHY4KZX326FX5CO6A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221310478 (562h30m39.092393513s) [noise=true , address=145.79.227.13:4802 , piece_id=QVC2GCHERYR55PIOWF43TQ34O3QFN5OWECBUBMUOJOUL36YCDIFA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221650125 (562h30m39.09273317s) [noise=true , address=45.91.116.14:4192 , piece_id=SLUTIZHU5ISKYGYROBUQBFDLCAGPV6XDXQTXHF5XDCCHOP46RQUQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221839355 (562h30m39.09292239s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.222138304 (562h30m39.093221339s) [noise=true , address=45.91.116.8:4048 , piece_id=3T6UZYUX7WEBWZCBNGE7OHYLRPIVH6YJPDBLMMRFTHKFUUQVUM5Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.222428887 (562h30m39.093511922s) [noise=true , address=145.79.227.50:4374 , piece_id=GVMZW5ZMBBPIVAYH5B3HUXWTLRAWIWTXSVMKZJHGI6UEQLWJRGRA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.227274724 (562h30m39.098357759s) [noise=true , address=145.79.227.11:4732 , piece_id=F5RNFOMZ4RYHFRLCO5LM4YLNOMA3SN3PM46RI7RSQCRF7J66WY2Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.229287257 (562h30m39.100370292s) [noise=true , address=145.79.227.2:4113 , piece_id=33NVLK2NBHERFZEFB47KOY5EPWKEB2MQ5WU6ENO6BE3H527O4ADA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.230594666 (562h30m39.101677701s) [noise=true , address=145.79.227.15:4876 , piece_id=7ZD3LEVP3DZEQZC7GLWU6BOPVZPAR542UURXGTXTBCDCMVBIRGLA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.231517844 (562h30m39.102600879s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.232631264 (562h30m39.103714299s) [noise=true , address=145.79.227.47:4265 , piece_id=KK5I5PQN6R5YEQTUQNDSERNV5UKRZ7URBSNCXYI5SLBSEJZRXGAQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.233220306 (562h30m39.104303341s) [noise=true , address=145.79.227.42:4123 , piece_id=KDB7MUWZXIS3QGGAPXLHWSD5ZM7WUSFRGV4AC5DMF5JWODAQ5VHA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.234069172 (562h30m39.105152217s) [noise=true , address=145.79.227.7:4293 , piece_id=IMTTUY5GYUTU5BZAZKXBXWLH3IFFZI3Z6TSUATJ3VLQHSDQJHI3Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.234955439 (562h30m39.106038504s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:40.937306407 (562h30m40.808389442s) [upload=493180]: Next returned nil segment Dec 29 01:59:40.937321756 (562h30m40.808404801s) [upload=493180]: Waiting for error group managing segments... Dec 29 01:59:41.217294798 (562h30m41.088377843s) [noise=true , address=145.79.227.15:4876 , piece_id=7ZD3LEVP3DZEQZC7GLWU6BOPVZPAR542UURXGTXTBCDCMVBIRGLA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.223671520 (562h30m41.094754555s) [noise=true , address=145.79.227.48:4301 , piece_id=5OEILVM4RJVX6IZSTNOAAJV3JWLUA7Z3TP6IP6TGUOSPGMMNCAJQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.227339554 (562h30m41.098422589s) [noise=true , address=145.79.227.2:4113 , piece_id=33NVLK2NBHERFZEFB47KOY5EPWKEB2MQ5WU6ENO6BE3H527O4ADA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.232457159 (562h30m41.103540204s) [noise=true , address=145.79.227.45:4231 , piece_id=22NRWH7WQKLRHFUZ42FHO7O7SR444WPD4EP3Y3KR6ZYL6RKDCL4A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.239167085 (562h30m41.11025013s) [noise=true , address=145.79.227.7:4293 , piece_id=IMTTUY5GYUTU5BZAZKXBXWLH3IFFZI3Z6TSUATJ3VLQHSDQJHI3Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.240513589 (562h30m41.111596634s) [noise=true , address=145.79.227.50:4374 , piece_id=GVMZW5ZMBBPIVAYH5B3HUXWTLRAWIWTXSVMKZJHGI6UEQLWJRGRA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.242912126 (562h30m41.113995172s) [noise=true , address=145.79.227.3:4150 , piece_id=ZJWIISMYPWPAIPV663EFAHEXO2W3FA67I36UHY4KZX326FX5CO6A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.247753995 (562h30m41.11883704s) [noise=true , address=145.79.227.42:4123 , piece_id=KDB7MUWZXIS3QGGAPXLHWSD5ZM7WUSFRGV4AC5DMF5JWODAQ5VHA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.252074161 (562h30m41.123157206s) [noise=true , address=145.79.227.13:4802 , piece_id=QVC2GCHERYR55PIOWF43TQ34O3QFN5OWECBUBMUOJOUL36YCDIFA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.253844002 (562h30m41.124927037s) [noise=true , address=145.79.227.11:4732 , piece_id=F5RNFOMZ4RYHFRLCO5LM4YLNOMA3SN3PM46RI7RSQCRF7J66WY2Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.271486786 (562h30m41.142569821s) [noise=true , address=145.79.227.47:4265 , piece_id=KK5I5PQN6R5YEQTUQNDSERNV5UKRZ7URBSNCXYI5SLBSEJZRXGAQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.454186376 (562h30m41.325269431s) [noise=true , address=45.91.116.8:4048 , piece_id=3T6UZYUX7WEBWZCBNGE7OHYLRPIVH6YJPDBLMMRFTHKFUUQVUM5Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.510719051 (562h30m41.381802096s) [noise=true , address=45.91.116.14:4192 , piece_id=SLUTIZHU5ISKYGYROBUQBFDLCAGPV6XDXQTXHF5XDCCHOP46RQUQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.510746734 (562h30m41.381829769s) [seg_pos={0 0}, upload=493180]: Segment reached optimal threshold of 13 pieces. Dec 29 01:59:41.511169308 (562h30m41.382252343s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511180940 (562h30m41.382263985s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511200658 (562h30m41.382283693s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511217840 (562h30m41.382300875s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511406069 (562h30m41.382489104s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511412521 (562h30m41.382495556s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511431707 (562h30m41.382514742s) [seg_pos={0 0}, upload=493180]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:59:41.511512171 (562h30m41.382595206s) [upload=493180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:59:41.511517241 (562h30m41.382600276s) [upload=493180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:59:41.801536229 (562h30m41.672619274s) [upload=493180]: Done uploading segments... Dec 29 01:59:41.801554483 (562h30m41.672637518s) [upload=493180]: Upload finished. err: <nil> Duration: 6.462958242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X......X....XXX..XX................XX....X.....X......X] duration: 30.004073868s trace: https://jaeger.dev.storj.io/trace/3e1f3039711db9f6 FAIL (2025-12-29 02:00:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 01:59:38.975293575 (562h30m38.84637661s) [upload=493180]: Starting upload Dec 29 01:59:38.975364220 (562h30m38.846447265s) [upload=493180]: Done starting upload Dec 29 01:59:38.975423523 (562h30m38.846506558s) [upload=493180]: Uploading segments... Dec 29 01:59:38.975540005 (562h30m38.84662305s) [upload=493180]: Got next segment. Inline: false Dec 29 01:59:38.975565974 (562h30m38.846649019s) [upload=493180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:59:38.975570132 (562h30m38.846653167s) [upload=493180]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:59:39.219880405 (562h30m39.09096344s) [seg_pos={0 0}, upload=493180]: Begin upload segment... Dec 29 01:59:39.219923637 (562h30m39.091006672s) [seg_pos={0 0}, upload=493180]: Done begin upload segment. Dec 29 01:59:39.219955067 (562h30m39.091038102s) [noise=true , address=145.79.227.45:4231 , piece_id=22NRWH7WQKLRHFUZ42FHO7O7SR444WPD4EP3Y3KR6ZYL6RKDCL4A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.219958904 (562h30m39.091041939s) [noise=true , address=145.79.227.48:4301 , piece_id=5OEILVM4RJVX6IZSTNOAAJV3JWLUA7Z3TP6IP6TGUOSPGMMNCAJQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.220464056 (562h30m39.091547091s) [noise=true , address=145.79.227.3:4150 , piece_id=ZJWIISMYPWPAIPV663EFAHEXO2W3FA67I36UHY4KZX326FX5CO6A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221310478 (562h30m39.092393513s) [noise=true , address=145.79.227.13:4802 , piece_id=QVC2GCHERYR55PIOWF43TQ34O3QFN5OWECBUBMUOJOUL36YCDIFA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221650125 (562h30m39.09273317s) [noise=true , address=45.91.116.14:4192 , piece_id=SLUTIZHU5ISKYGYROBUQBFDLCAGPV6XDXQTXHF5XDCCHOP46RQUQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.221839355 (562h30m39.09292239s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.222138304 (562h30m39.093221339s) [noise=true , address=45.91.116.8:4048 , piece_id=3T6UZYUX7WEBWZCBNGE7OHYLRPIVH6YJPDBLMMRFTHKFUUQVUM5Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.222428887 (562h30m39.093511922s) [noise=true , address=145.79.227.50:4374 , piece_id=GVMZW5ZMBBPIVAYH5B3HUXWTLRAWIWTXSVMKZJHGI6UEQLWJRGRA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.227274724 (562h30m39.098357759s) [noise=true , address=145.79.227.11:4732 , piece_id=F5RNFOMZ4RYHFRLCO5LM4YLNOMA3SN3PM46RI7RSQCRF7J66WY2Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.229287257 (562h30m39.100370292s) [noise=true , address=145.79.227.2:4113 , piece_id=33NVLK2NBHERFZEFB47KOY5EPWKEB2MQ5WU6ENO6BE3H527O4ADA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.230594666 (562h30m39.101677701s) [noise=true , address=145.79.227.15:4876 , piece_id=7ZD3LEVP3DZEQZC7GLWU6BOPVZPAR542UURXGTXTBCDCMVBIRGLA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.231517844 (562h30m39.102600879s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.232631264 (562h30m39.103714299s) [noise=true , address=145.79.227.47:4265 , piece_id=KK5I5PQN6R5YEQTUQNDSERNV5UKRZ7URBSNCXYI5SLBSEJZRXGAQ, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.233220306 (562h30m39.104303341s) [noise=true , address=145.79.227.42:4123 , piece_id=KDB7MUWZXIS3QGGAPXLHWSD5ZM7WUSFRGV4AC5DMF5JWODAQ5VHA, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.234069172 (562h30m39.105152217s) [noise=true , address=145.79.227.7:4293 , piece_id=IMTTUY5GYUTU5BZAZKXBXWLH3IFFZI3Z6TSUATJ3VLQHSDQJHI3Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:39.234955439 (562h30m39.106038504s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Uploading piece... Dec 29 01:59:40.937306407 (562h30m40.808389442s) [upload=493180]: Next returned nil segment Dec 29 01:59:40.937321756 (562h30m40.808404801s) [upload=493180]: Waiting for error group managing segments... Dec 29 01:59:41.217294798 (562h30m41.088377843s) [noise=true , address=145.79.227.15:4876 , piece_id=7ZD3LEVP3DZEQZC7GLWU6BOPVZPAR542UURXGTXTBCDCMVBIRGLA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.223671520 (562h30m41.094754555s) [noise=true , address=145.79.227.48:4301 , piece_id=5OEILVM4RJVX6IZSTNOAAJV3JWLUA7Z3TP6IP6TGUOSPGMMNCAJQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.227339554 (562h30m41.098422589s) [noise=true , address=145.79.227.2:4113 , piece_id=33NVLK2NBHERFZEFB47KOY5EPWKEB2MQ5WU6ENO6BE3H527O4ADA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.232457159 (562h30m41.103540204s) [noise=true , address=145.79.227.45:4231 , piece_id=22NRWH7WQKLRHFUZ42FHO7O7SR444WPD4EP3Y3KR6ZYL6RKDCL4A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.239167085 (562h30m41.11025013s) [noise=true , address=145.79.227.7:4293 , piece_id=IMTTUY5GYUTU5BZAZKXBXWLH3IFFZI3Z6TSUATJ3VLQHSDQJHI3Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.240513589 (562h30m41.111596634s) [noise=true , address=145.79.227.50:4374 , piece_id=GVMZW5ZMBBPIVAYH5B3HUXWTLRAWIWTXSVMKZJHGI6UEQLWJRGRA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.242912126 (562h30m41.113995172s) [noise=true , address=145.79.227.3:4150 , piece_id=ZJWIISMYPWPAIPV663EFAHEXO2W3FA67I36UHY4KZX326FX5CO6A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.247753995 (562h30m41.11883704s) [noise=true , address=145.79.227.42:4123 , piece_id=KDB7MUWZXIS3QGGAPXLHWSD5ZM7WUSFRGV4AC5DMF5JWODAQ5VHA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.252074161 (562h30m41.123157206s) [noise=true , address=145.79.227.13:4802 , piece_id=QVC2GCHERYR55PIOWF43TQ34O3QFN5OWECBUBMUOJOUL36YCDIFA, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.253844002 (562h30m41.124927037s) [noise=true , address=145.79.227.11:4732 , piece_id=F5RNFOMZ4RYHFRLCO5LM4YLNOMA3SN3PM46RI7RSQCRF7J66WY2Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.271486786 (562h30m41.142569821s) [noise=true , address=145.79.227.47:4265 , piece_id=KK5I5PQN6R5YEQTUQNDSERNV5UKRZ7URBSNCXYI5SLBSEJZRXGAQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.454186376 (562h30m41.325269431s) [noise=true , address=45.91.116.8:4048 , piece_id=3T6UZYUX7WEBWZCBNGE7OHYLRPIVH6YJPDBLMMRFTHKFUUQVUM5Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.510719051 (562h30m41.381802096s) [noise=true , address=45.91.116.14:4192 , piece_id=SLUTIZHU5ISKYGYROBUQBFDLCAGPV6XDXQTXHF5XDCCHOP46RQUQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: <nil> Dec 29 01:59:41.510746734 (562h30m41.381829769s) [seg_pos={0 0}, upload=493180]: Segment reached optimal threshold of 13 pieces. Dec 29 01:59:41.511169308 (562h30m41.382252343s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511180940 (562h30m41.382263985s) [noise=true , address=45.91.116.7:4011 , piece_id=5KJARLLLRJXR5C3TTDRMCZFQXCST6GOBQ6OXI4SEDDZ3R6PNG72Q, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511200658 (562h30m41.382283693s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511217840 (562h30m41.382300875s) [noise=true , address=45.91.116.9:4261 , piece_id=BGKSFQCJEUCGGKDULJJNMS3BJDW6YCOLMBPMDPFX3JKP66RCMTNQ, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511406069 (562h30m41.382489104s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: expected piece hash; context canceled; EOF Dec 29 01:59:41.511412521 (562h30m41.382495556s) [noise=true , address=45.91.116.6:4974 , piece_id=TU2M7LUQI7VNT6IGA7K56MFR6VLMONKPUFJIZJE52VHX525WNR6A, seg_pos={0 0}, upload=493180]: Long tail cancellation due to optimal threshold Dec 29 01:59:41.511431707 (562h30m41.382514742s) [seg_pos={0 0}, upload=493180]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:59:41.511512171 (562h30m41.382595206s) [upload=493180]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:59:41.511517241 (562h30m41.382600276s) [upload=493180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:59:41.801536229 (562h30m41.672619274s) [upload=493180]: Done uploading segments... Dec 29 01:59:41.801554483 (562h30m41.672637518s) [upload=493180]: Upload finished. err: <nil> Duration: 30.004098764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....X......X....XXX..XX................XX....X.....X......] duration: 6.417943952s trace: https://jaeger.dev.storj.io/trace/2dce4b39f86dacce FAIL (2025-12-29 01:59:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2h26m41.937755879s) [upload=493120]: Done starting upload Dec 29 01:55:42.066727559 (562h26m41.937810604s) [upload=493120]: Uploading segments... Dec 29 01:55:42.066944291 (562h26m41.938027336s) [upload=493120]: Got next segment. Inline: false Dec 29 01:55:42.067009175 (562h26m41.93809222s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:55:42.067017019 (562h26m41.938100055s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:55:42.313274721 (562h26m42.184357757s) [seg_pos={0 0}, upload=493120]: Begin upload segment... Dec 29 01:55:42.313323043 (562h26m42.184406078s) [seg_pos={0 0}, upload=493120]: Done begin upload segment. Dec 29 01:55:42.313349894 (562h26m42.18443293s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.313447551 (562h26m42.184530586s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.314687011 (562h26m42.185770046s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315389068 (562h26m42.186472103s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315889871 (562h26m42.186972906s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315944495 (562h26m42.187027531s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316312616 (562h26m42.187395651s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316628437 (562h26m42.187711472s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316834029 (562h26m42.187917064s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316954980 (562h26m42.188038015s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317274608 (562h26m42.188357643s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317470941 (562h26m42.188553976s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318115589 (562h26m42.189198634s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318835781 (562h26m42.189918816s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320111249 (562h26m42.191194284s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320989742 (562h26m42.192072777s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:44.033750189 (562h26m43.904833224s) [upload=493120]: Next returned nil segment Dec 29 01:55:44.033784184 (562h26m43.904867219s) [upload=493120]: Waiting for error group managing segments... Dec 29 01:55:44.320013127 (562h26m44.191096162s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.322332693 (562h26m44.193415728s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.325372531 (562h26m44.196455566s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.326138620 (562h26m44.197221665s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.341672238 (562h26m44.212755273s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.350355041 (562h26m44.221438076s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.356883152 (562h26m44.227966197s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.539135550 (562h26m44.410218585s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.542252296 (562h26m44.413335331s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.570562130 (562h26m44.441645165s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.662250505 (562h26m44.53333355s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.872221194 (562h26m44.743304229s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938863378 (562h26m44.809946413s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938883937 (562h26m44.809966972s) [seg_pos={0 0}, upload=493120]: Segment reached optimal threshold of 13 pieces. Dec 29 01:55:44.939034835 (562h26m44.81011787s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection Dec 29 01:55:44.939040315 (562h26m44.81012335s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939349173 (562h26m44.810432218s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 01:55:44.939360605 (562h26m44.81044364s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939696154 (562h26m44.810779189s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:56230->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 01:55:44.939704409 (562h26m44.810787445s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939723976 (562h26m44.810807021s) [seg_pos={0 0}, upload=493120]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:55:44.939793989 (562h26m44.810877025s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:55:44.939799731 (562h26m44.810882766s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:55:45.236501383 (562h26m45.107584428s) [upload=493120]: Done uploading segments... Dec 29 01:55:45.236523255 (562h26m45.10760629s) [upload=493120]: Upload finished. err: <nil> Duration: 6.418022431s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.....X......X....XXX..XX................XX....X.....X.....] duration: 5.825206771s trace: https://jaeger.dev.storj.io/trace/2dce4b39f86dacce FAIL (2025-12-29 01:58:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2h26m41.937755879s) [upload=493120]: Done starting upload Dec 29 01:55:42.066727559 (562h26m41.937810604s) [upload=493120]: Uploading segments... Dec 29 01:55:42.066944291 (562h26m41.938027336s) [upload=493120]: Got next segment. Inline: false Dec 29 01:55:42.067009175 (562h26m41.93809222s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:55:42.067017019 (562h26m41.938100055s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:55:42.313274721 (562h26m42.184357757s) [seg_pos={0 0}, upload=493120]: Begin upload segment... Dec 29 01:55:42.313323043 (562h26m42.184406078s) [seg_pos={0 0}, upload=493120]: Done begin upload segment. Dec 29 01:55:42.313349894 (562h26m42.18443293s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.313447551 (562h26m42.184530586s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.314687011 (562h26m42.185770046s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315389068 (562h26m42.186472103s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315889871 (562h26m42.186972906s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315944495 (562h26m42.187027531s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316312616 (562h26m42.187395651s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316628437 (562h26m42.187711472s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316834029 (562h26m42.187917064s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316954980 (562h26m42.188038015s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317274608 (562h26m42.188357643s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317470941 (562h26m42.188553976s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318115589 (562h26m42.189198634s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318835781 (562h26m42.189918816s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320111249 (562h26m42.191194284s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320989742 (562h26m42.192072777s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:44.033750189 (562h26m43.904833224s) [upload=493120]: Next returned nil segment Dec 29 01:55:44.033784184 (562h26m43.904867219s) [upload=493120]: Waiting for error group managing segments... Dec 29 01:55:44.320013127 (562h26m44.191096162s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.322332693 (562h26m44.193415728s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.325372531 (562h26m44.196455566s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.326138620 (562h26m44.197221665s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.341672238 (562h26m44.212755273s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.350355041 (562h26m44.221438076s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.356883152 (562h26m44.227966197s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.539135550 (562h26m44.410218585s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.542252296 (562h26m44.413335331s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.570562130 (562h26m44.441645165s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.662250505 (562h26m44.53333355s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.872221194 (562h26m44.743304229s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938863378 (562h26m44.809946413s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938883937 (562h26m44.809966972s) [seg_pos={0 0}, upload=493120]: Segment reached optimal threshold of 13 pieces. Dec 29 01:55:44.939034835 (562h26m44.81011787s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection Dec 29 01:55:44.939040315 (562h26m44.81012335s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939349173 (562h26m44.810432218s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 01:55:44.939360605 (562h26m44.81044364s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939696154 (562h26m44.810779189s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:56230->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 01:55:44.939704409 (562h26m44.810787445s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939723976 (562h26m44.810807021s) [seg_pos={0 0}, upload=493120]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:55:44.939793989 (562h26m44.810877025s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:55:44.939799731 (562h26m44.810882766s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:55:45.236501383 (562h26m45.107584428s) [upload=493120]: Done uploading segments... Dec 29 01:55:45.236523255 (562h26m45.10760629s) [upload=493120]: Upload finished. err: <nil> Duration: 5.825234495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X.....X......X....XXX..XX................XX....X.....X....] duration: 20.289805441s trace: https://jaeger.dev.storj.io/trace/2dce4b39f86dacce FAIL (2025-12-29 01:58:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2h26m41.937755879s) [upload=493120]: Done starting upload Dec 29 01:55:42.066727559 (562h26m41.937810604s) [upload=493120]: Uploading segments... Dec 29 01:55:42.066944291 (562h26m41.938027336s) [upload=493120]: Got next segment. Inline: false Dec 29 01:55:42.067009175 (562h26m41.93809222s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:55:42.067017019 (562h26m41.938100055s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:55:42.313274721 (562h26m42.184357757s) [seg_pos={0 0}, upload=493120]: Begin upload segment... Dec 29 01:55:42.313323043 (562h26m42.184406078s) [seg_pos={0 0}, upload=493120]: Done begin upload segment. Dec 29 01:55:42.313349894 (562h26m42.18443293s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.313447551 (562h26m42.184530586s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.314687011 (562h26m42.185770046s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315389068 (562h26m42.186472103s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315889871 (562h26m42.186972906s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315944495 (562h26m42.187027531s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316312616 (562h26m42.187395651s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316628437 (562h26m42.187711472s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316834029 (562h26m42.187917064s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316954980 (562h26m42.188038015s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317274608 (562h26m42.188357643s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317470941 (562h26m42.188553976s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318115589 (562h26m42.189198634s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318835781 (562h26m42.189918816s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320111249 (562h26m42.191194284s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320989742 (562h26m42.192072777s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:44.033750189 (562h26m43.904833224s) [upload=493120]: Next returned nil segment Dec 29 01:55:44.033784184 (562h26m43.904867219s) [upload=493120]: Waiting for error group managing segments... Dec 29 01:55:44.320013127 (562h26m44.191096162s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.322332693 (562h26m44.193415728s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.325372531 (562h26m44.196455566s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.326138620 (562h26m44.197221665s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.341672238 (562h26m44.212755273s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.350355041 (562h26m44.221438076s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.356883152 (562h26m44.227966197s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.539135550 (562h26m44.410218585s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.542252296 (562h26m44.413335331s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.570562130 (562h26m44.441645165s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.662250505 (562h26m44.53333355s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.872221194 (562h26m44.743304229s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938863378 (562h26m44.809946413s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938883937 (562h26m44.809966972s) [seg_pos={0 0}, upload=493120]: Segment reached optimal threshold of 13 pieces. Dec 29 01:55:44.939034835 (562h26m44.81011787s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection Dec 29 01:55:44.939040315 (562h26m44.81012335s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939349173 (562h26m44.810432218s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 01:55:44.939360605 (562h26m44.81044364s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939696154 (562h26m44.810779189s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:56230->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 01:55:44.939704409 (562h26m44.810787445s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939723976 (562h26m44.810807021s) [seg_pos={0 0}, upload=493120]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:55:44.939793989 (562h26m44.810877025s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:55:44.939799731 (562h26m44.810882766s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:55:45.236501383 (562h26m45.107584428s) [upload=493120]: Done uploading segments... Dec 29 01:55:45.236523255 (562h26m45.10760629s) [upload=493120]: Upload finished. err: <nil> Duration: 20.289850377s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.X.....X......X....XXX..XX................XX....X.....X...] duration: 6.081688572s trace: https://jaeger.dev.storj.io/trace/2dce4b39f86dacce FAIL (2025-12-29 01:57:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2h26m41.937755879s) [upload=493120]: Done starting upload Dec 29 01:55:42.066727559 (562h26m41.937810604s) [upload=493120]: Uploading segments... Dec 29 01:55:42.066944291 (562h26m41.938027336s) [upload=493120]: Got next segment. Inline: false Dec 29 01:55:42.067009175 (562h26m41.93809222s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:55:42.067017019 (562h26m41.938100055s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:55:42.313274721 (562h26m42.184357757s) [seg_pos={0 0}, upload=493120]: Begin upload segment... Dec 29 01:55:42.313323043 (562h26m42.184406078s) [seg_pos={0 0}, upload=493120]: Done begin upload segment. Dec 29 01:55:42.313349894 (562h26m42.18443293s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.313447551 (562h26m42.184530586s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.314687011 (562h26m42.185770046s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315389068 (562h26m42.186472103s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315889871 (562h26m42.186972906s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315944495 (562h26m42.187027531s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316312616 (562h26m42.187395651s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316628437 (562h26m42.187711472s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316834029 (562h26m42.187917064s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316954980 (562h26m42.188038015s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317274608 (562h26m42.188357643s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317470941 (562h26m42.188553976s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318115589 (562h26m42.189198634s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318835781 (562h26m42.189918816s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320111249 (562h26m42.191194284s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320989742 (562h26m42.192072777s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:44.033750189 (562h26m43.904833224s) [upload=493120]: Next returned nil segment Dec 29 01:55:44.033784184 (562h26m43.904867219s) [upload=493120]: Waiting for error group managing segments... Dec 29 01:55:44.320013127 (562h26m44.191096162s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.322332693 (562h26m44.193415728s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.325372531 (562h26m44.196455566s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.326138620 (562h26m44.197221665s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.341672238 (562h26m44.212755273s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.350355041 (562h26m44.221438076s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.356883152 (562h26m44.227966197s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.539135550 (562h26m44.410218585s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.542252296 (562h26m44.413335331s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.570562130 (562h26m44.441645165s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.662250505 (562h26m44.53333355s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.872221194 (562h26m44.743304229s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938863378 (562h26m44.809946413s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938883937 (562h26m44.809966972s) [seg_pos={0 0}, upload=493120]: Segment reached optimal threshold of 13 pieces. Dec 29 01:55:44.939034835 (562h26m44.81011787s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection Dec 29 01:55:44.939040315 (562h26m44.81012335s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939349173 (562h26m44.810432218s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 01:55:44.939360605 (562h26m44.81044364s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939696154 (562h26m44.810779189s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:56230->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 01:55:44.939704409 (562h26m44.810787445s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939723976 (562h26m44.810807021s) [seg_pos={0 0}, upload=493120]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:55:44.939793989 (562h26m44.810877025s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:55:44.939799731 (562h26m44.810882766s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:55:45.236501383 (562h26m45.107584428s) [upload=493120]: Done uploading segments... Dec 29 01:55:45.236523255 (562h26m45.10760629s) [upload=493120]: Upload finished. err: <nil> Duration: 6.081715463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X.....X......X....XXX..XX................XX....X.....X..] duration: 5.857292167s trace: https://jaeger.dev.storj.io/trace/2dce4b39f86dacce FAIL (2025-12-29 01:57:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2h26m41.937755879s) [upload=493120]: Done starting upload Dec 29 01:55:42.066727559 (562h26m41.937810604s) [upload=493120]: Uploading segments... Dec 29 01:55:42.066944291 (562h26m41.938027336s) [upload=493120]: Got next segment. Inline: false Dec 29 01:55:42.067009175 (562h26m41.93809222s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:55:42.067017019 (562h26m41.938100055s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:55:42.313274721 (562h26m42.184357757s) [seg_pos={0 0}, upload=493120]: Begin upload segment... Dec 29 01:55:42.313323043 (562h26m42.184406078s) [seg_pos={0 0}, upload=493120]: Done begin upload segment. Dec 29 01:55:42.313349894 (562h26m42.18443293s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.313447551 (562h26m42.184530586s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.314687011 (562h26m42.185770046s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315389068 (562h26m42.186472103s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315889871 (562h26m42.186972906s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315944495 (562h26m42.187027531s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316312616 (562h26m42.187395651s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316628437 (562h26m42.187711472s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316834029 (562h26m42.187917064s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316954980 (562h26m42.188038015s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317274608 (562h26m42.188357643s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317470941 (562h26m42.188553976s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318115589 (562h26m42.189198634s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318835781 (562h26m42.189918816s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320111249 (562h26m42.191194284s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320989742 (562h26m42.192072777s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:44.033750189 (562h26m43.904833224s) [upload=493120]: Next returned nil segment Dec 29 01:55:44.033784184 (562h26m43.904867219s) [upload=493120]: Waiting for error group managing segments... Dec 29 01:55:44.320013127 (562h26m44.191096162s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.322332693 (562h26m44.193415728s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.325372531 (562h26m44.196455566s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.326138620 (562h26m44.197221665s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.341672238 (562h26m44.212755273s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.350355041 (562h26m44.221438076s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.356883152 (562h26m44.227966197s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.539135550 (562h26m44.410218585s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.542252296 (562h26m44.413335331s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.570562130 (562h26m44.441645165s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.662250505 (562h26m44.53333355s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.872221194 (562h26m44.743304229s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938863378 (562h26m44.809946413s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938883937 (562h26m44.809966972s) [seg_pos={0 0}, upload=493120]: Segment reached optimal threshold of 13 pieces. Dec 29 01:55:44.939034835 (562h26m44.81011787s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection Dec 29 01:55:44.939040315 (562h26m44.81012335s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939349173 (562h26m44.810432218s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 01:55:44.939360605 (562h26m44.81044364s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939696154 (562h26m44.810779189s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:56230->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 01:55:44.939704409 (562h26m44.810787445s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939723976 (562h26m44.810807021s) [seg_pos={0 0}, upload=493120]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:55:44.939793989 (562h26m44.810877025s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:55:44.939799731 (562h26m44.810882766s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:55:45.236501383 (562h26m45.107584428s) [upload=493120]: Done uploading segments... Dec 29 01:55:45.236523255 (562h26m45.10760629s) [upload=493120]: Upload finished. err: <nil> Duration: 5.857315802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.X.....X......X....XXX..XX................XX....X.....X.] duration: 5.883213934s trace: https://jaeger.dev.storj.io/trace/2dce4b39f86dacce FAIL (2025-12-29 01:56:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2h26m41.937755879s) [upload=493120]: Done starting upload Dec 29 01:55:42.066727559 (562h26m41.937810604s) [upload=493120]: Uploading segments... Dec 29 01:55:42.066944291 (562h26m41.938027336s) [upload=493120]: Got next segment. Inline: false Dec 29 01:55:42.067009175 (562h26m41.93809222s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:55:42.067017019 (562h26m41.938100055s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:55:42.313274721 (562h26m42.184357757s) [seg_pos={0 0}, upload=493120]: Begin upload segment... Dec 29 01:55:42.313323043 (562h26m42.184406078s) [seg_pos={0 0}, upload=493120]: Done begin upload segment. Dec 29 01:55:42.313349894 (562h26m42.18443293s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.313447551 (562h26m42.184530586s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.314687011 (562h26m42.185770046s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315389068 (562h26m42.186472103s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315889871 (562h26m42.186972906s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315944495 (562h26m42.187027531s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316312616 (562h26m42.187395651s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316628437 (562h26m42.187711472s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316834029 (562h26m42.187917064s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316954980 (562h26m42.188038015s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317274608 (562h26m42.188357643s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317470941 (562h26m42.188553976s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318115589 (562h26m42.189198634s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318835781 (562h26m42.189918816s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320111249 (562h26m42.191194284s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320989742 (562h26m42.192072777s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:44.033750189 (562h26m43.904833224s) [upload=493120]: Next returned nil segment Dec 29 01:55:44.033784184 (562h26m43.904867219s) [upload=493120]: Waiting for error group managing segments... Dec 29 01:55:44.320013127 (562h26m44.191096162s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.322332693 (562h26m44.193415728s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.325372531 (562h26m44.196455566s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.326138620 (562h26m44.197221665s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.341672238 (562h26m44.212755273s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.350355041 (562h26m44.221438076s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.356883152 (562h26m44.227966197s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.539135550 (562h26m44.410218585s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.542252296 (562h26m44.413335331s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.570562130 (562h26m44.441645165s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.662250505 (562h26m44.53333355s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.872221194 (562h26m44.743304229s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938863378 (562h26m44.809946413s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938883937 (562h26m44.809966972s) [seg_pos={0 0}, upload=493120]: Segment reached optimal threshold of 13 pieces. Dec 29 01:55:44.939034835 (562h26m44.81011787s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection Dec 29 01:55:44.939040315 (562h26m44.81012335s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939349173 (562h26m44.810432218s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 01:55:44.939360605 (562h26m44.81044364s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939696154 (562h26m44.810779189s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:56230->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 01:55:44.939704409 (562h26m44.810787445s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939723976 (562h26m44.810807021s) [seg_pos={0 0}, upload=493120]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:55:44.939793989 (562h26m44.810877025s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:55:44.939799731 (562h26m44.810882766s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:55:45.236501383 (562h26m45.107584428s) [upload=493120]: Done uploading segments... Dec 29 01:55:45.236523255 (562h26m45.10760629s) [upload=493120]: Upload finished. err: <nil> Duration: 5.883237449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.X.....X......X....XXX..XX................XX....X.....X] duration: 30.002730445s trace: https://jaeger.dev.storj.io/trace/2dce4b39f86dacce FAIL (2025-12-29 01:56:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2h26m41.937755879s) [upload=493120]: Done starting upload Dec 29 01:55:42.066727559 (562h26m41.937810604s) [upload=493120]: Uploading segments... Dec 29 01:55:42.066944291 (562h26m41.938027336s) [upload=493120]: Got next segment. Inline: false Dec 29 01:55:42.067009175 (562h26m41.93809222s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:55:42.067017019 (562h26m41.938100055s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:55:42.313274721 (562h26m42.184357757s) [seg_pos={0 0}, upload=493120]: Begin upload segment... Dec 29 01:55:42.313323043 (562h26m42.184406078s) [seg_pos={0 0}, upload=493120]: Done begin upload segment. Dec 29 01:55:42.313349894 (562h26m42.18443293s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.313447551 (562h26m42.184530586s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.314687011 (562h26m42.185770046s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315389068 (562h26m42.186472103s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315889871 (562h26m42.186972906s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.315944495 (562h26m42.187027531s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316312616 (562h26m42.187395651s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316628437 (562h26m42.187711472s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316834029 (562h26m42.187917064s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.316954980 (562h26m42.188038015s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317274608 (562h26m42.188357643s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.317470941 (562h26m42.188553976s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318115589 (562h26m42.189198634s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.318835781 (562h26m42.189918816s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320111249 (562h26m42.191194284s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:42.320989742 (562h26m42.192072777s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Uploading piece... Dec 29 01:55:44.033750189 (562h26m43.904833224s) [upload=493120]: Next returned nil segment Dec 29 01:55:44.033784184 (562h26m43.904867219s) [upload=493120]: Waiting for error group managing segments... Dec 29 01:55:44.320013127 (562h26m44.191096162s) [noise=true , address=145.79.227.11:4730 , piece_id=DABJ3CMLGZOSSIVDI52NNWWY7Z6WG4BA3K6L77LORGVWANBBNN6Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.322332693 (562h26m44.193415728s) [noise=true , address=145.79.227.19:4077 , piece_id=SCNFQEW7DJWIN7SRURYVVGLQH75EYW5FL55F4K4NANN2JUCS267Q, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.325372531 (562h26m44.196455566s) [noise=true , address=145.79.227.41:4085 , piece_id=NOT5RLZR4DPXP6PPZ64EPXIR4EFGXIBZ52FN2ZEE7YZJMBJZHYPA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.326138620 (562h26m44.197221665s) [noise=true , address=145.79.227.5:4223 , piece_id=EKKJLUEAZOHDEPZ3TNTPEVQLPMMFSUJVE45EG2FVXCEVVQ7PDMVA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.341672238 (562h26m44.212755273s) [noise=true , address=145.79.227.24:4582 , piece_id=E2MBTM4EEDJP4KQYGBGFAKUGYX2CKGSAX2OALMU7RO4BWIXMSHEQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.350355041 (562h26m44.221438076s) [noise=true , address=145.79.227.21:4473 , piece_id=TIQIPQTUUOXAX3TH5ZWJA627WSEHI2BI4Z5SD4Z2Z6RCEQSLKLOQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.356883152 (562h26m44.227966197s) [noise=true , address=145.79.227.13:4803 , piece_id=5G5LMFKO6NXUSHBNNJK3SZALZGDPDYHZXAFPXNDSZOITWVC4NBBQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.539135550 (562h26m44.410218585s) [noise=true , address=77.90.31.18:4805 , piece_id=D4QJUCEIZRXUHYOBK2ALYJVMAM3QMFNHODA6KE6D4KUXGU66AAAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.542252296 (562h26m44.413335331s) [noise=true , address=45.91.116.11:4117 , piece_id=B7QN7SVCNFLQBFW6Q57VKGKYETPVVQDUZEJ5YNKMTF27Q6KGWEAA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.570562130 (562h26m44.441645165s) [noise=true , address=77.90.31.14:4661 , piece_id=H7WNAGRMTHPTJL5ZICER62HA2RUQU7RJ3ZUXEWZOZ75DTARBOIEA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.662250505 (562h26m44.53333355s) [noise=true , address=77.90.31.10:4519 , piece_id=SOQ7B47GF6ORD67Y3MANBXOHXOWHQOFINZZ2P3EJNUUB4KVFUZHQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.872221194 (562h26m44.743304229s) [noise=true , address=45.91.116.9:4262 , piece_id=OTSSNGSPXIGID2XS44EBBPUIG5XIB7QQOFWF5MQIOTW4PHKZTILQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938863378 (562h26m44.809946413s) [noise=true , address=45.91.116.8:4047 , piece_id=HPD5WBKFOS3EODIISD5JHST6SEVOGHWPAJ5GZ5LB6WE6A3GEM5EQ, seg_pos={0 0}, upload=493120]: Done uploading piece. err: <nil> Dec 29 01:55:44.938883937 (562h26m44.809966972s) [seg_pos={0 0}, upload=493120]: Segment reached optimal threshold of 13 pieces. Dec 29 01:55:44.939034835 (562h26m44.81011787s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR): protocol: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34784->145.79.227.16:4912: use of closed network connection Dec 29 01:55:44.939040315 (562h26m44.81012335s) [noise=true , address=145.79.227.16:4912 , piece_id=PRHU7JUGLRK2KVQXKLVUMIEXMYO6VO6FMGFSH7OZRHSZEETGEKHA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939349173 (562h26m44.810432218s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 01:55:44.939360605 (562h26m44.81044364s) [noise=true , address=145.79.227.43:4159 , piece_id=2FVWH3EGC5HDSJIJ7MDF7GMQ4IU6C6KP7M6UWZYZ572F3G3QRBSA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939696154 (562h26m44.810779189s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: read tcp 188.166.225.170:56230->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 01:55:44.939704409 (562h26m44.810787445s) [noise=true , address=45.91.116.6:4973 , piece_id=J5HB3LBVWUVTEZY4JKZN36TOYXA64QQOHBTCGBNYH2K26TSIB5RA, seg_pos={0 0}, upload=493120]: Long tail cancellation due to optimal threshold Dec 29 01:55:44.939723976 (562h26m44.810807021s) [seg_pos={0 0}, upload=493120]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:55:44.939793989 (562h26m44.810877025s) [upload=493120]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:55:44.939799731 (562h26m44.810882766s) [upload=493120]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:55:45.236501383 (562h26m45.107584428s) [upload=493120]: Done uploading segments... Dec 29 01:55:45.236523255 (562h26m45.10760629s) [upload=493120]: Upload finished. err: <nil> Duration: 30.002750753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.X.....X......X....XXX..XX................XX....X.....] duration: 5.828633136s trace: https://jaeger.dev.storj.io/trace/2b1a9e306f4890e8 FAIL (2025-12-29 01:55:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:29.793602575 (562h22m29.664685611s) [upload=493060]: Starting upload Dec 29 01:51:29.793663692 (562h22m29.664746727s) [upload=493060]: Done starting upload Dec 29 01:51:29.793730840 (562h22m29.664813875s) [upload=493060]: Uploading segments... Dec 29 01:51:29.793976919 (562h22m29.665059964s) [upload=493060]: Got next segment. Inline: false Dec 29 01:51:29.794021253 (562h22m29.665104288s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:51:29.794028737 (562h22m29.665111772s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:51:30.037620874 (562h22m29.908703909s) [seg_pos={0 0}, upload=493060]: Begin upload segment... Dec 29 01:51:30.037665358 (562h22m29.908748393s) [seg_pos={0 0}, upload=493060]: Done begin upload segment. Dec 29 01:51:30.037708100 (562h22m29.908791135s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.037792420 (562h22m29.908875465s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038327288 (562h22m29.909410334s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038791353 (562h22m29.909874398s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038876414 (562h22m29.909959449s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039167980 (562h22m29.910251015s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039418216 (562h22m29.910501251s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.042280276 (562h22m29.913363311s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043554872 (562h22m29.914637918s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043641007 (562h22m29.914724042s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044704822 (562h22m29.915787867s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044821425 (562h22m29.91590447s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046127972 (562h22m29.917211007s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046135556 (562h22m29.917218591s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.058854665 (562h22m29.92993771s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.060493927 (562h22m29.931576962s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:31.758588509 (562h22m31.629671544s) [upload=493060]: Next returned nil segment Dec 29 01:51:31.758601966 (562h22m31.629685011s) [upload=493060]: Waiting for error group managing segments... Dec 29 01:51:32.043231442 (562h22m31.914314487s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.048678504 (562h22m31.919761549s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.061186611 (562h22m31.932269656s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.073081260 (562h22m31.944164295s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.074180552 (562h22m31.945263597s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.075956153 (562h22m31.947039198s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.078911401 (562h22m31.949994436s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.081766507 (562h22m31.952849542s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.082309611 (562h22m31.953392646s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.095734233 (562h22m31.966817268s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.283778720 (562h22m32.154861765s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.349872180 (562h22m32.220955215s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363385652 (562h22m32.234468687s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363410028 (562h22m32.234493073s) [seg_pos={0 0}, upload=493060]: Segment reached optimal threshold of 13 pieces. Dec 29 01:51:32.363592846 (562h22m32.234675891s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363598537 (562h22m32.234681572s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363715019 (562h22m32.234798054s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection Dec 29 01:51:32.363720178 (562h22m32.234803213s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363726971 (562h22m32.234810016s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363743753 (562h22m32.234826788s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363773731 (562h22m32.234856766s) [seg_pos={0 0}, upload=493060]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:51:32.363866807 (562h22m32.234949852s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:51:32.363872990 (562h22m32.234956035s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:51:32.662054151 (562h22m32.533137197s) [upload=493060]: Done uploading segments... Dec 29 01:51:32.662071865 (562h22m32.5331549s) [upload=493060]: Upload finished. err: <nil> Duration: 5.82865606s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.....X.X.....X......X....XXX..XX................XX....X....] duration: 13.339424672s trace: https://jaeger.dev.storj.io/trace/2b1a9e306f4890e8 FAIL (2025-12-29 01:54:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:29.793602575 (562h22m29.664685611s) [upload=493060]: Starting upload Dec 29 01:51:29.793663692 (562h22m29.664746727s) [upload=493060]: Done starting upload Dec 29 01:51:29.793730840 (562h22m29.664813875s) [upload=493060]: Uploading segments... Dec 29 01:51:29.793976919 (562h22m29.665059964s) [upload=493060]: Got next segment. Inline: false Dec 29 01:51:29.794021253 (562h22m29.665104288s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:51:29.794028737 (562h22m29.665111772s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:51:30.037620874 (562h22m29.908703909s) [seg_pos={0 0}, upload=493060]: Begin upload segment... Dec 29 01:51:30.037665358 (562h22m29.908748393s) [seg_pos={0 0}, upload=493060]: Done begin upload segment. Dec 29 01:51:30.037708100 (562h22m29.908791135s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.037792420 (562h22m29.908875465s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038327288 (562h22m29.909410334s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038791353 (562h22m29.909874398s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038876414 (562h22m29.909959449s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039167980 (562h22m29.910251015s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039418216 (562h22m29.910501251s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.042280276 (562h22m29.913363311s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043554872 (562h22m29.914637918s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043641007 (562h22m29.914724042s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044704822 (562h22m29.915787867s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044821425 (562h22m29.91590447s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046127972 (562h22m29.917211007s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046135556 (562h22m29.917218591s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.058854665 (562h22m29.92993771s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.060493927 (562h22m29.931576962s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:31.758588509 (562h22m31.629671544s) [upload=493060]: Next returned nil segment Dec 29 01:51:31.758601966 (562h22m31.629685011s) [upload=493060]: Waiting for error group managing segments... Dec 29 01:51:32.043231442 (562h22m31.914314487s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.048678504 (562h22m31.919761549s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.061186611 (562h22m31.932269656s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.073081260 (562h22m31.944164295s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.074180552 (562h22m31.945263597s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.075956153 (562h22m31.947039198s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.078911401 (562h22m31.949994436s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.081766507 (562h22m31.952849542s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.082309611 (562h22m31.953392646s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.095734233 (562h22m31.966817268s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.283778720 (562h22m32.154861765s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.349872180 (562h22m32.220955215s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363385652 (562h22m32.234468687s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363410028 (562h22m32.234493073s) [seg_pos={0 0}, upload=493060]: Segment reached optimal threshold of 13 pieces. Dec 29 01:51:32.363592846 (562h22m32.234675891s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363598537 (562h22m32.234681572s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363715019 (562h22m32.234798054s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection Dec 29 01:51:32.363720178 (562h22m32.234803213s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363726971 (562h22m32.234810016s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363743753 (562h22m32.234826788s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363773731 (562h22m32.234856766s) [seg_pos={0 0}, upload=493060]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:51:32.363866807 (562h22m32.234949852s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:51:32.363872990 (562h22m32.234956035s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:51:32.662054151 (562h22m32.533137197s) [upload=493060]: Done uploading segments... Dec 29 01:51:32.662071865 (562h22m32.5331549s) [upload=493060]: Upload finished. err: <nil> Duration: 13.339453646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.....X.X.....X......X....XXX..XX................XX....X...] duration: 28.710586118s trace: https://jaeger.dev.storj.io/trace/2b1a9e306f4890e8 FAIL (2025-12-29 01:53:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:29.793602575 (562h22m29.664685611s) [upload=493060]: Starting upload Dec 29 01:51:29.793663692 (562h22m29.664746727s) [upload=493060]: Done starting upload Dec 29 01:51:29.793730840 (562h22m29.664813875s) [upload=493060]: Uploading segments... Dec 29 01:51:29.793976919 (562h22m29.665059964s) [upload=493060]: Got next segment. Inline: false Dec 29 01:51:29.794021253 (562h22m29.665104288s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:51:29.794028737 (562h22m29.665111772s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:51:30.037620874 (562h22m29.908703909s) [seg_pos={0 0}, upload=493060]: Begin upload segment... Dec 29 01:51:30.037665358 (562h22m29.908748393s) [seg_pos={0 0}, upload=493060]: Done begin upload segment. Dec 29 01:51:30.037708100 (562h22m29.908791135s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.037792420 (562h22m29.908875465s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038327288 (562h22m29.909410334s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038791353 (562h22m29.909874398s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038876414 (562h22m29.909959449s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039167980 (562h22m29.910251015s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039418216 (562h22m29.910501251s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.042280276 (562h22m29.913363311s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043554872 (562h22m29.914637918s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043641007 (562h22m29.914724042s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044704822 (562h22m29.915787867s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044821425 (562h22m29.91590447s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046127972 (562h22m29.917211007s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046135556 (562h22m29.917218591s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.058854665 (562h22m29.92993771s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.060493927 (562h22m29.931576962s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:31.758588509 (562h22m31.629671544s) [upload=493060]: Next returned nil segment Dec 29 01:51:31.758601966 (562h22m31.629685011s) [upload=493060]: Waiting for error group managing segments... Dec 29 01:51:32.043231442 (562h22m31.914314487s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.048678504 (562h22m31.919761549s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.061186611 (562h22m31.932269656s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.073081260 (562h22m31.944164295s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.074180552 (562h22m31.945263597s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.075956153 (562h22m31.947039198s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.078911401 (562h22m31.949994436s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.081766507 (562h22m31.952849542s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.082309611 (562h22m31.953392646s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.095734233 (562h22m31.966817268s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.283778720 (562h22m32.154861765s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.349872180 (562h22m32.220955215s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363385652 (562h22m32.234468687s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363410028 (562h22m32.234493073s) [seg_pos={0 0}, upload=493060]: Segment reached optimal threshold of 13 pieces. Dec 29 01:51:32.363592846 (562h22m32.234675891s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363598537 (562h22m32.234681572s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363715019 (562h22m32.234798054s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection Dec 29 01:51:32.363720178 (562h22m32.234803213s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363726971 (562h22m32.234810016s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363743753 (562h22m32.234826788s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363773731 (562h22m32.234856766s) [seg_pos={0 0}, upload=493060]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:51:32.363866807 (562h22m32.234949852s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:51:32.363872990 (562h22m32.234956035s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:51:32.662054151 (562h22m32.533137197s) [upload=493060]: Done uploading segments... Dec 29 01:51:32.662071865 (562h22m32.5331549s) [upload=493060]: Upload finished. err: <nil> Duration: 28.710609752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.....X.X.....X......X....XXX..XX................XX....X..] duration: 21.946525666s trace: https://jaeger.dev.storj.io/trace/2b1a9e306f4890e8 FAIL (2025-12-29 01:53:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:29.793602575 (562h22m29.664685611s) [upload=493060]: Starting upload Dec 29 01:51:29.793663692 (562h22m29.664746727s) [upload=493060]: Done starting upload Dec 29 01:51:29.793730840 (562h22m29.664813875s) [upload=493060]: Uploading segments... Dec 29 01:51:29.793976919 (562h22m29.665059964s) [upload=493060]: Got next segment. Inline: false Dec 29 01:51:29.794021253 (562h22m29.665104288s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:51:29.794028737 (562h22m29.665111772s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:51:30.037620874 (562h22m29.908703909s) [seg_pos={0 0}, upload=493060]: Begin upload segment... Dec 29 01:51:30.037665358 (562h22m29.908748393s) [seg_pos={0 0}, upload=493060]: Done begin upload segment. Dec 29 01:51:30.037708100 (562h22m29.908791135s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.037792420 (562h22m29.908875465s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038327288 (562h22m29.909410334s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038791353 (562h22m29.909874398s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038876414 (562h22m29.909959449s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039167980 (562h22m29.910251015s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039418216 (562h22m29.910501251s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.042280276 (562h22m29.913363311s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043554872 (562h22m29.914637918s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043641007 (562h22m29.914724042s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044704822 (562h22m29.915787867s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044821425 (562h22m29.91590447s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046127972 (562h22m29.917211007s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046135556 (562h22m29.917218591s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.058854665 (562h22m29.92993771s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.060493927 (562h22m29.931576962s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:31.758588509 (562h22m31.629671544s) [upload=493060]: Next returned nil segment Dec 29 01:51:31.758601966 (562h22m31.629685011s) [upload=493060]: Waiting for error group managing segments... Dec 29 01:51:32.043231442 (562h22m31.914314487s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.048678504 (562h22m31.919761549s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.061186611 (562h22m31.932269656s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.073081260 (562h22m31.944164295s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.074180552 (562h22m31.945263597s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.075956153 (562h22m31.947039198s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.078911401 (562h22m31.949994436s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.081766507 (562h22m31.952849542s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.082309611 (562h22m31.953392646s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.095734233 (562h22m31.966817268s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.283778720 (562h22m32.154861765s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.349872180 (562h22m32.220955215s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363385652 (562h22m32.234468687s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363410028 (562h22m32.234493073s) [seg_pos={0 0}, upload=493060]: Segment reached optimal threshold of 13 pieces. Dec 29 01:51:32.363592846 (562h22m32.234675891s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363598537 (562h22m32.234681572s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363715019 (562h22m32.234798054s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection Dec 29 01:51:32.363720178 (562h22m32.234803213s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363726971 (562h22m32.234810016s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363743753 (562h22m32.234826788s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363773731 (562h22m32.234856766s) [seg_pos={0 0}, upload=493060]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:51:32.363866807 (562h22m32.234949852s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:51:32.363872990 (562h22m32.234956035s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:51:32.662054151 (562h22m32.533137197s) [upload=493060]: Done uploading segments... Dec 29 01:51:32.662071865 (562h22m32.5331549s) [upload=493060]: Upload finished. err: <nil> Duration: 21.946556415s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.....X.X.....X......X....XXX..XX................XX....X.] duration: 6.059917247s trace: https://jaeger.dev.storj.io/trace/2b1a9e306f4890e8 FAIL (2025-12-29 01:52:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:29.793602575 (562h22m29.664685611s) [upload=493060]: Starting upload Dec 29 01:51:29.793663692 (562h22m29.664746727s) [upload=493060]: Done starting upload Dec 29 01:51:29.793730840 (562h22m29.664813875s) [upload=493060]: Uploading segments... Dec 29 01:51:29.793976919 (562h22m29.665059964s) [upload=493060]: Got next segment. Inline: false Dec 29 01:51:29.794021253 (562h22m29.665104288s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:51:29.794028737 (562h22m29.665111772s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:51:30.037620874 (562h22m29.908703909s) [seg_pos={0 0}, upload=493060]: Begin upload segment... Dec 29 01:51:30.037665358 (562h22m29.908748393s) [seg_pos={0 0}, upload=493060]: Done begin upload segment. Dec 29 01:51:30.037708100 (562h22m29.908791135s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.037792420 (562h22m29.908875465s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038327288 (562h22m29.909410334s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038791353 (562h22m29.909874398s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038876414 (562h22m29.909959449s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039167980 (562h22m29.910251015s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039418216 (562h22m29.910501251s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.042280276 (562h22m29.913363311s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043554872 (562h22m29.914637918s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043641007 (562h22m29.914724042s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044704822 (562h22m29.915787867s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044821425 (562h22m29.91590447s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046127972 (562h22m29.917211007s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046135556 (562h22m29.917218591s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.058854665 (562h22m29.92993771s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.060493927 (562h22m29.931576962s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:31.758588509 (562h22m31.629671544s) [upload=493060]: Next returned nil segment Dec 29 01:51:31.758601966 (562h22m31.629685011s) [upload=493060]: Waiting for error group managing segments... Dec 29 01:51:32.043231442 (562h22m31.914314487s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.048678504 (562h22m31.919761549s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.061186611 (562h22m31.932269656s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.073081260 (562h22m31.944164295s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.074180552 (562h22m31.945263597s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.075956153 (562h22m31.947039198s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.078911401 (562h22m31.949994436s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.081766507 (562h22m31.952849542s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.082309611 (562h22m31.953392646s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.095734233 (562h22m31.966817268s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.283778720 (562h22m32.154861765s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.349872180 (562h22m32.220955215s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363385652 (562h22m32.234468687s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363410028 (562h22m32.234493073s) [seg_pos={0 0}, upload=493060]: Segment reached optimal threshold of 13 pieces. Dec 29 01:51:32.363592846 (562h22m32.234675891s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363598537 (562h22m32.234681572s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363715019 (562h22m32.234798054s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection Dec 29 01:51:32.363720178 (562h22m32.234803213s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363726971 (562h22m32.234810016s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363743753 (562h22m32.234826788s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363773731 (562h22m32.234856766s) [seg_pos={0 0}, upload=493060]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:51:32.363866807 (562h22m32.234949852s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:51:32.363872990 (562h22m32.234956035s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:51:32.662054151 (562h22m32.533137197s) [upload=493060]: Done uploading segments... Dec 29 01:51:32.662071865 (562h22m32.5331549s) [upload=493060]: Upload finished. err: <nil> Duration: 6.059945581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.....X.X.....X......X....XXX..XX................XX....X] duration: 30.002880754s trace: https://jaeger.dev.storj.io/trace/2b1a9e306f4890e8 FAIL (2025-12-29 01:51:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1:29.793602575 (562h22m29.664685611s) [upload=493060]: Starting upload Dec 29 01:51:29.793663692 (562h22m29.664746727s) [upload=493060]: Done starting upload Dec 29 01:51:29.793730840 (562h22m29.664813875s) [upload=493060]: Uploading segments... Dec 29 01:51:29.793976919 (562h22m29.665059964s) [upload=493060]: Got next segment. Inline: false Dec 29 01:51:29.794021253 (562h22m29.665104288s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:51:29.794028737 (562h22m29.665111772s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:51:30.037620874 (562h22m29.908703909s) [seg_pos={0 0}, upload=493060]: Begin upload segment... Dec 29 01:51:30.037665358 (562h22m29.908748393s) [seg_pos={0 0}, upload=493060]: Done begin upload segment. Dec 29 01:51:30.037708100 (562h22m29.908791135s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.037792420 (562h22m29.908875465s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038327288 (562h22m29.909410334s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038791353 (562h22m29.909874398s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.038876414 (562h22m29.909959449s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039167980 (562h22m29.910251015s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.039418216 (562h22m29.910501251s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.042280276 (562h22m29.913363311s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043554872 (562h22m29.914637918s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.043641007 (562h22m29.914724042s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044704822 (562h22m29.915787867s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.044821425 (562h22m29.91590447s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046127972 (562h22m29.917211007s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.046135556 (562h22m29.917218591s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.058854665 (562h22m29.92993771s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:30.060493927 (562h22m29.931576962s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Uploading piece... Dec 29 01:51:31.758588509 (562h22m31.629671544s) [upload=493060]: Next returned nil segment Dec 29 01:51:31.758601966 (562h22m31.629685011s) [upload=493060]: Waiting for error group managing segments... Dec 29 01:51:32.043231442 (562h22m31.914314487s) [noise=true , address=145.79.227.50:4375 , piece_id=PTWFQLOYCX2PHGDKDZXTZ5SPVW7CGBDGPB4YRFLI7JEEAQRNTH7Q, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.048678504 (562h22m31.919761549s) [noise=true , address=145.79.227.45:4231 , piece_id=IIXXU2A2DENEAALLTUJNO27UVHCQZLDTGUVP5WZOB2Y4TXCW5EIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.061186611 (562h22m31.932269656s) [noise=true , address=145.79.227.46:4602 , piece_id=DMHSFZEZS2JOODSTPPAVRW6OVRND6C2F7RPRV24ZVNB7JEBD2MVA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.073081260 (562h22m31.944164295s) [noise=true , address=145.79.227.7:4293 , piece_id=366355573BWORR4JKNOMHKC4KGUMRP7YYAC5XY2AO6I42O67GPTA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.074180552 (562h22m31.945263597s) [noise=true , address=145.79.227.26:4654 , piece_id=OVJYA2ZC6ONRFCHFC6CX5TLKW62YXFNE5UE3KGLLINUGTWD2GTLQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.075956153 (562h22m31.947039198s) [noise=true , address=145.79.227.11:4732 , piece_id=43VC6TITQKF2JGV74ULLDQJ2NZM5WCO3VOWN6YMVLN6SO7AUQYLA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.078911401 (562h22m31.949994436s) [noise=true , address=145.79.227.38:4979 , piece_id=NATDG7GFVDQR3JEKUC6LWBRJLS7P2JGF4RUNYBPMELTBSE3DB7RQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.081766507 (562h22m31.952849542s) [noise=true , address=145.79.227.14:4839 , piece_id=F7EED2PUAEVP4TWOF53M5WAAGAYH6YNFW5HEAKMQEVB7U732FOBA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.082309611 (562h22m31.953392646s) [noise=true , address=145.79.227.16:4910 , piece_id=OAD6XZ7E7AD5EAXJSPNUPPXERYVEI3X4YXWHNSCKMCFBY5YHAMSA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.095734233 (562h22m31.966817268s) [noise=true , address=145.79.227.5:4223 , piece_id=IMKVP5I6J4TR6PVS2QC3EVXXV5AAEGVOMOB6U65CDJJ6IDZP6JUQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.283778720 (562h22m32.154861765s) [noise=true , address=77.90.31.21:4878 , piece_id=UBPRYMSVLPPYRCIX67KMCIPCKKJ373PH66CXMANTCEL6LRYOMFIA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.349872180 (562h22m32.220955215s) [noise=true , address=77.90.31.16:4735 , piece_id=WATWTN7KKXSWUM5BDD6X6PM7WTEH5IWDT5UXVTNTA6UJ3AHXEZFA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363385652 (562h22m32.234468687s) [noise=true , address=45.91.116.5:4299 , piece_id=EIGQEPLILSZU7VN5WUJYSJKV45QD63QXCSEC4ZNMDATHSTZLK5CQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: <nil> Dec 29 01:51:32.363410028 (562h22m32.234493073s) [seg_pos={0 0}, upload=493060]: Segment reached optimal threshold of 13 pieces. Dec 29 01:51:32.363592846 (562h22m32.234675891s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363598537 (562h22m32.234681572s) [noise=true , address=45.91.116.9:4264 , piece_id=DO7TK6WFU7F6TIBDDUC2JSFUCQBTUTZIRWIKKQ5N4F5MXIRGXGFQ, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363715019 (562h22m32.234798054s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58708->145.79.227.19:4079: use of closed network connection Dec 29 01:51:32.363720178 (562h22m32.234803213s) [noise=true , address=145.79.227.19:4079 , piece_id=E74YUNSX7DXUSOWK6NO5VBUICAVDFZBQB6HBG7EPIYWXCISG2ILA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363726971 (562h22m32.234810016s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 01:51:32.363743753 (562h22m32.234826788s) [noise=true , address=145.79.227.44:4194 , piece_id=6FRHAAOTZO562GNWETYDYW53H7QA66TZYGEZ2BR5JJBSJPW26XUA, seg_pos={0 0}, upload=493060]: Long tail cancellation due to optimal threshold Dec 29 01:51:32.363773731 (562h22m32.234856766s) [seg_pos={0 0}, upload=493060]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:51:32.363866807 (562h22m32.234949852s) [upload=493060]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:51:32.363872990 (562h22m32.234956035s) [upload=493060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:51:32.662054151 (562h22m32.533137197s) [upload=493060]: Done uploading segments... Dec 29 01:51:32.662071865 (562h22m32.5331549s) [upload=493060]: Upload finished. err: <nil> Duration: 30.00289927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X.....X.X.....X......X....XXX..XX................XX....] duration: 6.435397023s trace: https://jaeger.dev.storj.io/trace/2064d9f473aab404 FAIL (2025-12-29 01:51:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 29 01:48:43.267520061 (562h19m43.138603097s) [upload=493016]: Got next segment. Inline: false Dec 29 01:48:43.267548946 (562h19m43.138631991s) [upload=493016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:48:43.267553986 (562h19m43.138637031s) [upload=493016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:48:43.511601399 (562h19m43.382684434s) [seg_pos={0 0}, upload=493016]: Begin upload segment... Dec 29 01:48:43.511653848 (562h19m43.382736883s) [seg_pos={0 0}, upload=493016]: Done begin upload segment. Dec 29 01:48:43.511752927 (562h19m43.382835962s) [noise=true , address=77.90.31.13:4627 , piece_id=S7FWUE7SQK4TTHU2O6E6XXDOPXCDVABCBEOPOAJUTPWPDLKPWPBQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.511757966 (562h19m43.382841002s) [noise=true , address=45.91.116.13:4156 , piece_id=LU3IOKYGW47IBGWIFFNITWLF2ZNUYY3JDKJDWL3IL5RAEMR2W5LA, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.512998028 (562h19m43.384081063s) [noise=true , address=45.91.116.14:4192 , piece_id=QAAHEUKHR25WYDUW4RAJ2BEFB3ZYAWPEGWH6ABVRVYVC4ZXW2JLQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.513133266 (562h19m43.384216301s) [noise=true , address=145.79.227.41:4085 , piece_id=4RFCLZXHBALJABRMNMZK5PNFHY63SDNRHUNQPTMIBDIXGMVZAP6A, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.513495094 (562h19m43.384578129s) [noise=true , address=77.90.31.21:4877 , piece_id=VJQBLUHX3UV2YU4OAGPMEGXZ4YAMSDQT6MDKTM3ZG446N7D34NZQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.513777071 (562h19m43.384860106s) [noise=true , address=145.79.227.42:4121 , piece_id=PDTDXOCCEAZQPNMJF2GY5DSSZBOGVDVATAUYOBH7UXUP3KLYPZ7Q, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.514139231 (562h19m43.385222266s) [noise=true , address=45.91.116.10:4082 , piece_id=ZQZG2FV2XDBMNMH5BD2HCEVTUOR4PVNC64XUWJFJXZIM4ZS3IVHQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.514210277 (562h19m43.385293312s) [noise=true , address=45.91.116.5:4299 , piece_id=EQPT5664IBUMKCTRXEVVR4KCNBO24L65A46VBPFSTHDOM7VTUQZA, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.514551456 (562h19m43.385634492s) [noise=true , address=45.91.116.4:4939 , piece_id=GURVXTMXVBEITWJDZ4TMJWVA4V7HHIYJCOWHRSNZIQCBHACHMS5Q, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.514986575 (562h19m43.38606962s) [noise=true , address=45.91.116.7:4011 , piece_id=7WKY662GIPXGWEAPFI7O6BZ67N7HYWGTP6UONNVBVTMMG7XJYKKQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.515378190 (562h19m43.386461235s) [noise=true , address=77.90.31.10:4517 , piece_id=4GTBIMRYIGZQTKMTRZ4MQMLLNEFQJSA2PF5MGDYM2SWBQ7GLVLHQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.515689363 (562h19m43.386772408s) [noise=true , address=145.79.227.50:4375 , piece_id=JTMALWGVXEILAQQBFMMD3SYUCBWPCVYYAH4UEGSV3HS6SKTLUO5Q, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.516016456 (562h19m43.387099491s) [noise=true , address=145.79.227.14:4838 , piece_id=YQI435BEK5DFXDYAAZKHIFZ4XZF5E6LPZ4PRGR7PTG23UMJ62VMQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.516473196 (562h19m43.387556241s) [noise=true , address=77.90.31.18:4805 , piece_id=UNHPUXOQMQVGWVLWUQFCESEPMUPMLVOJOBSW7KYE6GQZDVK3AEXQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.516560742 (562h19m43.387643777s) [noise=true , address=145.79.227.52:4446 , piece_id=IXF6ZTNUTPK2Q2OMBXGTFCR6HLTFLTXZ7HR44AIL5JGLAHZCTJJQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.517309769 (562h19m43.388392814s) [noise=true , address=145.79.227.46:4603 , piece_id=5J6PN5LULRLYJCATI3OO5OHJ2HICB5XZCGDMXOBF5PSFV3BQLFHA, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:45.199169767 (562h19m45.070252812s) [upload=493016]: Next returned nil segment Dec 29 01:48:45.199195296 (562h19m45.070278331s) [upload=493016]: Waiting for error group managing segments... Dec 29 01:48:45.506169099 (562h19m45.377252134s) [noise=true , address=145.79.227.50:4375 , piece_id=JTMALWGVXEILAQQBFMMD3SYUCBWPCVYYAH4UEGSV3HS6SKTLUO5Q, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.515076860 (562h19m45.386159895s) [noise=true , address=145.79.227.42:4121 , piece_id=PDTDXOCCEAZQPNMJF2GY5DSSZBOGVDVATAUYOBH7UXUP3KLYPZ7Q, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.517676140 (562h19m45.388759175s) [noise=true , address=145.79.227.46:4603 , piece_id=5J6PN5LULRLYJCATI3OO5OHJ2HICB5XZCGDMXOBF5PSFV3BQLFHA, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.530241395 (562h19m45.40132443s) [noise=true , address=145.79.227.41:4085 , piece_id=4RFCLZXHBALJABRMNMZK5PNFHY63SDNRHUNQPTMIBDIXGMVZAP6A, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.552514546 (562h19m45.423597591s) [noise=true , address=145.79.227.52:4446 , piece_id=IXF6ZTNUTPK2Q2OMBXGTFCR6HLTFLTXZ7HR44AIL5JGLAHZCTJJQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.572145565 (562h19m45.4432286s) [noise=true , address=145.79.227.14:4838 , piece_id=YQI435BEK5DFXDYAAZKHIFZ4XZF5E6LPZ4PRGR7PTG23UMJ62VMQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.688329112 (562h19m45.559412157s) [noise=true , address=77.90.31.21:4877 , piece_id=VJQBLUHX3UV2YU4OAGPMEGXZ4YAMSDQT6MDKTM3ZG446N7D34NZQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.698790782 (562h19m45.569873817s) [noise=true , address=77.90.31.18:4805 , piece_id=UNHPUXOQMQVGWVLWUQFCESEPMUPMLVOJOBSW7KYE6GQZDVK3AEXQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.764176144 (562h19m45.635259179s) [noise=true , address=45.91.116.5:4299 , piece_id=EQPT5664IBUMKCTRXEVVR4KCNBO24L65A46VBPFSTHDOM7VTUQZA, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.788149562 (562h19m45.659232757s) [noise=true , address=77.90.31.10:4517 , piece_id=4GTBIMRYIGZQTKMTRZ4MQMLLNEFQJSA2PF5MGDYM2SWBQ7GLVLHQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.799616627 (562h19m45.670699662s) [noise=true , address=77.90.31.13:4627 , piece_id=S7FWUE7SQK4TTHU2O6E6XXDOPXCDVABCBEOPOAJUTPWPDLKPWPBQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.859517905 (562h19m45.73060094s) [noise=true , address=45.91.116.13:4156 , piece_id=LU3IOKYGW47IBGWIFFNITWLF2ZNUYY3JDKJDWL3IL5RAEMR2W5LA, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.921419563 (562h19m45.792502598s) [noise=true , address=45.91.116.14:4192 , piece_id=QAAHEUKHR25WYDUW4RAJ2BEFB3ZYAWPEGWH6ABVRVYVC4ZXW2JLQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.921437387 (562h19m45.792520422s) [seg_pos={0 0}, upload=493016]: Segment reached optimal threshold of 13 pieces. Dec 29 01:48:45.921579588 (562h19m45.792662623s) [noise=true , address=45.91.116.4:4939 , piece_id=GURVXTMXVBEITWJDZ4TMJWVA4V7HHIYJCOWHRSNZIQCBHACHMS5Q, seg_pos={0 0}, upload=493016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 01:48:45.921585238 (562h19m45.792668273s) [noise=true , address=45.91.116.4:4939 , piece_id=GURVXTMXVBEITWJDZ4TMJWVA4V7HHIYJCOWHRSNZIQCBHACHMS5Q, seg_pos={0 0}, upload=493016]: Long tail cancellation due to optimal threshold Dec 29 01:48:45.921727389 (562h19m45.792810434s) [noise=true , address=45.91.116.10:4082 , piece_id=ZQZG2FV2XDBMNMH5BD2HCEVTUOR4PVNC64XUWJFJXZIM4ZS3IVHQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:36428->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:36428->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36428->45.91.116.10:4082: use of closed network connection Dec 29 01:48:45.921737589 (562h19m45.792820634s) [noise=true , address=45.91.116.10:4082 , piece_id=ZQZG2FV2XDBMNMH5BD2HCEVTUOR4PVNC64XUWJFJXZIM4ZS3IVHQ, seg_pos={0 0}, upload=493016]: Long tail cancellation due to optimal threshold Dec 29 01:48:45.921822891 (562h19m45.792905936s) [noise=true , address=45.91.116.7:4011 , piece_id=7WKY662GIPXGWEAPFI7O6BZ67N7HYWGTP6UONNVBVTMMG7XJYKKQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:43434->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43434->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43434->45.91.116.7:4011: use of closed network connection Dec 29 01:48:45.921832870 (562h19m45.792915905s) [noise=true , address=45.91.116.7:4011 , piece_id=7WKY662GIPXGWEAPFI7O6BZ67N7HYWGTP6UONNVBVTMMG7XJYKKQ, seg_pos={0 0}, upload=493016]: Long tail cancellation due to optimal threshold Dec 29 01:48:45.921860332 (562h19m45.792943377s) [seg_pos={0 0}, upload=493016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:48:45.921961385 (562h19m45.79304443s) [upload=493016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:48:45.921968147 (562h19m45.793051182s) [upload=493016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:48:46.217617497 (562h19m46.088700542s) [upload=493016]: Done uploading segments... Dec 29 01:48:46.217644718 (562h19m46.088727763s) [upload=493016]: Upload finished. err: <nil> Duration: 6.435430558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X.....X.X.....X......X....XXX..XX................XX...] duration: 7.387276519s trace: https://jaeger.dev.storj.io/trace/2064d9f473aab404 FAIL (2025-12-29 01:50:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 29 01:48:43.267520061 (562h19m43.138603097s) [upload=493016]: Got next segment. Inline: false Dec 29 01:48:43.267548946 (562h19m43.138631991s) [upload=493016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 01:48:43.267553986 (562h19m43.138637031s) [upload=493016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 01:48:43.511601399 (562h19m43.382684434s) [seg_pos={0 0}, upload=493016]: Begin upload segment... Dec 29 01:48:43.511653848 (562h19m43.382736883s) [seg_pos={0 0}, upload=493016]: Done begin upload segment. Dec 29 01:48:43.511752927 (562h19m43.382835962s) [noise=true , address=77.90.31.13:4627 , piece_id=S7FWUE7SQK4TTHU2O6E6XXDOPXCDVABCBEOPOAJUTPWPDLKPWPBQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.511757966 (562h19m43.382841002s) [noise=true , address=45.91.116.13:4156 , piece_id=LU3IOKYGW47IBGWIFFNITWLF2ZNUYY3JDKJDWL3IL5RAEMR2W5LA, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.512998028 (562h19m43.384081063s) [noise=true , address=45.91.116.14:4192 , piece_id=QAAHEUKHR25WYDUW4RAJ2BEFB3ZYAWPEGWH6ABVRVYVC4ZXW2JLQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.513133266 (562h19m43.384216301s) [noise=true , address=145.79.227.41:4085 , piece_id=4RFCLZXHBALJABRMNMZK5PNFHY63SDNRHUNQPTMIBDIXGMVZAP6A, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.513495094 (562h19m43.384578129s) [noise=true , address=77.90.31.21:4877 , piece_id=VJQBLUHX3UV2YU4OAGPMEGXZ4YAMSDQT6MDKTM3ZG446N7D34NZQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.513777071 (562h19m43.384860106s) [noise=true , address=145.79.227.42:4121 , piece_id=PDTDXOCCEAZQPNMJF2GY5DSSZBOGVDVATAUYOBH7UXUP3KLYPZ7Q, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.514139231 (562h19m43.385222266s) [noise=true , address=45.91.116.10:4082 , piece_id=ZQZG2FV2XDBMNMH5BD2HCEVTUOR4PVNC64XUWJFJXZIM4ZS3IVHQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.514210277 (562h19m43.385293312s) [noise=true , address=45.91.116.5:4299 , piece_id=EQPT5664IBUMKCTRXEVVR4KCNBO24L65A46VBPFSTHDOM7VTUQZA, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.514551456 (562h19m43.385634492s) [noise=true , address=45.91.116.4:4939 , piece_id=GURVXTMXVBEITWJDZ4TMJWVA4V7HHIYJCOWHRSNZIQCBHACHMS5Q, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.514986575 (562h19m43.38606962s) [noise=true , address=45.91.116.7:4011 , piece_id=7WKY662GIPXGWEAPFI7O6BZ67N7HYWGTP6UONNVBVTMMG7XJYKKQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.515378190 (562h19m43.386461235s) [noise=true , address=77.90.31.10:4517 , piece_id=4GTBIMRYIGZQTKMTRZ4MQMLLNEFQJSA2PF5MGDYM2SWBQ7GLVLHQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.515689363 (562h19m43.386772408s) [noise=true , address=145.79.227.50:4375 , piece_id=JTMALWGVXEILAQQBFMMD3SYUCBWPCVYYAH4UEGSV3HS6SKTLUO5Q, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.516016456 (562h19m43.387099491s) [noise=true , address=145.79.227.14:4838 , piece_id=YQI435BEK5DFXDYAAZKHIFZ4XZF5E6LPZ4PRGR7PTG23UMJ62VMQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.516473196 (562h19m43.387556241s) [noise=true , address=77.90.31.18:4805 , piece_id=UNHPUXOQMQVGWVLWUQFCESEPMUPMLVOJOBSW7KYE6GQZDVK3AEXQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.516560742 (562h19m43.387643777s) [noise=true , address=145.79.227.52:4446 , piece_id=IXF6ZTNUTPK2Q2OMBXGTFCR6HLTFLTXZ7HR44AIL5JGLAHZCTJJQ, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:43.517309769 (562h19m43.388392814s) [noise=true , address=145.79.227.46:4603 , piece_id=5J6PN5LULRLYJCATI3OO5OHJ2HICB5XZCGDMXOBF5PSFV3BQLFHA, seg_pos={0 0}, upload=493016]: Uploading piece... Dec 29 01:48:45.199169767 (562h19m45.070252812s) [upload=493016]: Next returned nil segment Dec 29 01:48:45.199195296 (562h19m45.070278331s) [upload=493016]: Waiting for error group managing segments... Dec 29 01:48:45.506169099 (562h19m45.377252134s) [noise=true , address=145.79.227.50:4375 , piece_id=JTMALWGVXEILAQQBFMMD3SYUCBWPCVYYAH4UEGSV3HS6SKTLUO5Q, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.515076860 (562h19m45.386159895s) [noise=true , address=145.79.227.42:4121 , piece_id=PDTDXOCCEAZQPNMJF2GY5DSSZBOGVDVATAUYOBH7UXUP3KLYPZ7Q, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.517676140 (562h19m45.388759175s) [noise=true , address=145.79.227.46:4603 , piece_id=5J6PN5LULRLYJCATI3OO5OHJ2HICB5XZCGDMXOBF5PSFV3BQLFHA, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.530241395 (562h19m45.40132443s) [noise=true , address=145.79.227.41:4085 , piece_id=4RFCLZXHBALJABRMNMZK5PNFHY63SDNRHUNQPTMIBDIXGMVZAP6A, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.552514546 (562h19m45.423597591s) [noise=true , address=145.79.227.52:4446 , piece_id=IXF6ZTNUTPK2Q2OMBXGTFCR6HLTFLTXZ7HR44AIL5JGLAHZCTJJQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.572145565 (562h19m45.4432286s) [noise=true , address=145.79.227.14:4838 , piece_id=YQI435BEK5DFXDYAAZKHIFZ4XZF5E6LPZ4PRGR7PTG23UMJ62VMQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.688329112 (562h19m45.559412157s) [noise=true , address=77.90.31.21:4877 , piece_id=VJQBLUHX3UV2YU4OAGPMEGXZ4YAMSDQT6MDKTM3ZG446N7D34NZQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.698790782 (562h19m45.569873817s) [noise=true , address=77.90.31.18:4805 , piece_id=UNHPUXOQMQVGWVLWUQFCESEPMUPMLVOJOBSW7KYE6GQZDVK3AEXQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.764176144 (562h19m45.635259179s) [noise=true , address=45.91.116.5:4299 , piece_id=EQPT5664IBUMKCTRXEVVR4KCNBO24L65A46VBPFSTHDOM7VTUQZA, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.788149562 (562h19m45.659232757s) [noise=true , address=77.90.31.10:4517 , piece_id=4GTBIMRYIGZQTKMTRZ4MQMLLNEFQJSA2PF5MGDYM2SWBQ7GLVLHQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.799616627 (562h19m45.670699662s) [noise=true , address=77.90.31.13:4627 , piece_id=S7FWUE7SQK4TTHU2O6E6XXDOPXCDVABCBEOPOAJUTPWPDLKPWPBQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.859517905 (562h19m45.73060094s) [noise=true , address=45.91.116.13:4156 , piece_id=LU3IOKYGW47IBGWIFFNITWLF2ZNUYY3JDKJDWL3IL5RAEMR2W5LA, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.921419563 (562h19m45.792502598s) [noise=true , address=45.91.116.14:4192 , piece_id=QAAHEUKHR25WYDUW4RAJ2BEFB3ZYAWPEGWH6ABVRVYVC4ZXW2JLQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: <nil> Dec 29 01:48:45.921437387 (562h19m45.792520422s) [seg_pos={0 0}, upload=493016]: Segment reached optimal threshold of 13 pieces. Dec 29 01:48:45.921579588 (562h19m45.792662623s) [noise=true , address=45.91.116.4:4939 , piece_id=GURVXTMXVBEITWJDZ4TMJWVA4V7HHIYJCOWHRSNZIQCBHACHMS5Q, seg_pos={0 0}, upload=493016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 01:48:45.921585238 (562h19m45.792668273s) [noise=true , address=45.91.116.4:4939 , piece_id=GURVXTMXVBEITWJDZ4TMJWVA4V7HHIYJCOWHRSNZIQCBHACHMS5Q, seg_pos={0 0}, upload=493016]: Long tail cancellation due to optimal threshold Dec 29 01:48:45.921727389 (562h19m45.792810434s) [noise=true , address=45.91.116.10:4082 , piece_id=ZQZG2FV2XDBMNMH5BD2HCEVTUOR4PVNC64XUWJFJXZIM4ZS3IVHQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:36428->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:36428->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36428->45.91.116.10:4082: use of closed network connection Dec 29 01:48:45.921737589 (562h19m45.792820634s) [noise=true , address=45.91.116.10:4082 , piece_id=ZQZG2FV2XDBMNMH5BD2HCEVTUOR4PVNC64XUWJFJXZIM4ZS3IVHQ, seg_pos={0 0}, upload=493016]: Long tail cancellation due to optimal threshold Dec 29 01:48:45.921822891 (562h19m45.792905936s) [noise=true , address=45.91.116.7:4011 , piece_id=7WKY662GIPXGWEAPFI7O6BZ67N7HYWGTP6UONNVBVTMMG7XJYKKQ, seg_pos={0 0}, upload=493016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:43434->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:43434->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43434->45.91.116.7:4011: use of closed network connection Dec 29 01:48:45.921832870 (562h19m45.792915905s) [noise=true , address=45.91.116.7:4011 , piece_id=7WKY662GIPXGWEAPFI7O6BZ67N7HYWGTP6UONNVBVTMMG7XJYKKQ, seg_pos={0 0}, upload=493016]: Long tail cancellation due to optimal threshold Dec 29 01:48:45.921860332 (562h19m45.792943377s) [seg_pos={0 0}, upload=493016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 01:48:45.921961385 (562h19m45.79304443s) [upload=493016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 01:48:45.921968147 (562h19m45.793051182s) [upload=493016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 01:48:46.217617497 (562h19m46.088700542s) [upload=493016]: Done uploading segments... Dec 29 01:48:46.217644718 (562h19m46.088727763s) [upload=493016]: Upload finished. err: <nil> Duration: 7.387305154s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)