Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.817339907s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:50:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.817369622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.753124445s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:50:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.753153321s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.904236922s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:50:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.904266599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.975970117s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:50:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.976007228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.207658703s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:50:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.207692837s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.045202714s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:49:54 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 4.04523788s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.832428526s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:49:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.832461949s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.723614595s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:49:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.723640704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 3.645692935s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:49:29 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.645722732s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.957398184s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:49:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.957430406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.324441409s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:48:54 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.324468139s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 3.893881478s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:48:47 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.893904662s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.127556554s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:48:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.127602321s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.850944154s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:48:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.850973078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.997086094s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:48:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.997114618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.253312727s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:48:09 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.253415343s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.902636077s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:47:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.9026693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.904578454s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:47:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.904606016s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 3.55681855s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:47:43 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.55684984s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.901977851s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:47:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.902002337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.971489555s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:47:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.971511156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.478973649s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:47:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.479002303s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.937935191s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:47:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.937964348s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.985413184s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:46:58 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.985431187s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.979051014s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:46:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.979076641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.839967114s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:46:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.839994926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.298826063s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:46:20 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.298855891s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 3.788427696s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:46:14 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.788457193s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.774673379s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:46:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.774702435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.417689095s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:45:46 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.417717209s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.999641565s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:45:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.999671963s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 3.562690111s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:45:36 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.562725078s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.879725871s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:45:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.879761759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 4.01537704s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:45:08 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 4.015411026s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.751240035s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:45:07 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.751275572s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.847325372s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:44:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.847351992s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.42062355s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:44:33 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.420647716s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.049522702s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:44:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.049553701s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.042203458s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:44:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.042234427s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 3.770918307s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:43:59 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.770942764s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.917995953s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:43:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.918026683s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.03289893s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:43:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 6.032924899s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.437450067s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:43:22 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.437553514s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 3.860545184s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:42:59 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.860573947s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.851827847s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:42:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.851853586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.965670003s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:42:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.965722863s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 3.85632152s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:42:39 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.856355435s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.922547161s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:42:20 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.922579031s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.029000462s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:42:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.029048042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.926403933s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:41:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.926433869s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)