Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 3.392367836s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:34:57 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.392397523s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.978859799s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:34:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.978885528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.150222051s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:34:50 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.150253752s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.710138194s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:34:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.710178782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.09121335s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:34:20 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.091249069s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.971209077s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:34:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.971238784s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 3.513692661s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:33:58 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.513725995s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.851699061s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:33:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.851728568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.924927453s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:33:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.924961397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.151268618s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:33:26 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.15129676s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.023232392s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:33:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.023261909s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 3.7545084s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:32:47 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.754555149s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.925802858s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:32:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.925836102s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.833421767s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:32:30 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.833457876s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.915194929s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:31:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.915215437s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 3.635548988s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:31:54 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.635584426s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.818715364s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:31:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.818743087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.021086857s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:31:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.021130229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.468208637s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:31:26 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.468268281s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.677492922s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:31:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.677526598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 3.753937603s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:31:15 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.753969312s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.298024927s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:31:08 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.298059133s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 3.517642032s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:30:18 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.517674593s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.02228025s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:30:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.022388335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.023684072s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:30:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 6.023706955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.149447107s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:30:11 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.149480791s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 3.912542503s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:29:49 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.912580867s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.922306402s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:29:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.92239477s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.760738876s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:29:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.76076797s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.283714538s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:28:57 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.283745036s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.974430425s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:28:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.974456584s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 3.871475405s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:28:47 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.871504801s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.852509765s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:28:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.852539341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.086777545s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:28:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.086806149s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.165411808s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:27:53 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.165620547s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 3.869401646s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:27:37 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.869437505s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.86289495s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:27:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.862927642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.733390317s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:27:17 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.733419401s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.832049116s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:27:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.832076087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.449469977s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:27:00 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.449501827s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 3.856578046s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:26:52 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.856618033s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.842801339s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:26:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.842828609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.372723227s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:26:33 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.372750619s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.929742267s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:26:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.929768799s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.938030053s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:26:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.938061363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.56700323s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:25:52 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.567034871s core uplink us1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 3.705654913s trace: https://jaeger.dev.storj.io/trace/1826fc12eb97ad6d FAIL (2025-11-19 11:25:41 UTC) orbiter warning: operation: sync2.Copy uplink: piecestore: (Node ID: 12VpHQ9agBBm65AWhdMWSPDNzqJyiHsPYfQV1KDL1eMv1hk8cwN, Piece ID: JEJRXJ4ODOL3JXRRX3KNXHLNSEFJJWKBMMAB5TP2HSKEYOWRYZLQ) context deadline exceeded; piecestore: (Node ID: 128bohZvc8pUuRBLDosNFjtmirJeXZnAeFmLS8Uf67xAcv1i6dQ, Piece ID: UWWNJVCFWPTZRE4RRJEOVALYKEW3EXACJJULMVUDSTJ7OCCPGURA) context deadline exceeded; piecestore: (Node ID: 12gGjjeKkFD1uMfqccQCTJnQoUZ8CztSvFYottRkghGrbAhwxqj, Piece ID: XQSUQFZSCKJV45MBXZGZ52KO3QR3HNFR3W6X3A7G7NWGCXQ62VPQ) context deadline exceeded; piecestore: (Node ID: 1qMvhA9gakN3pyukerUWue2TfBeLiBTEDDWULCxdN5LcBFaSdn, Piece ID: F4Z74T5CUL5L37SGJLZJ354FNCRY22YN3LHWXMR3CSD3B6QG7Y2A) context deadline exceeded; piecestore: (Node ID: 12dn15Yr9N1nA1STdMKmX2dVLNKeTHshDaqAWzGF8gb59u1K9Ve, Piece ID: B6X3KT4VKEKUPFDT3EMKNCF5ZFWYBSPADUYTWZ2SM7TZJJZYNBZA) context deadline exceeded; piecestore: (Node ID: 1WPr7TRD3As3kHeYcsgzJuyzkNN7ktxumophSqbfPveHnyq9y8, Piece ID: MKA4DODJT6BB2VEPFMQGM4VYXPI7ZWOVTVL2Q5VBO3OSLFHIAI2A) context deadline exceeded; piecestore: (Node ID: 12NzWsxJe1qKVubZPwzkF6XorMp35Ri4nra1hUMkdeG4p3z7h4s, Piece ID: FUVVLL3R3BJRBEFLC5TXFZN3MDJG3GO5MWCXG4Q4ZFELN6OW3CFQ) context deadline exceeded; piecestore: (Node ID: 1gVnpQu7bWogHfd8KrAmytkaNHeUHykcBWuSpk5TbMy86moqju, Piece ID: WM3XVPJWEGP674YDPL3MEA36TBS6KG4HZZBC2QLZ3FFVK36JFXZQ) context deadline exceeded; piecestore: (Node ID: 12oBKFEUdCh7UkuZsesS4e43L18ocZTv9FeZwr1vznuEWPDETQP, Piece ID: 344IJDH33LDZ5AQWY4Y5BB5SDH5ALHPYWZSQY27NVCC4ZWUDKVTQ) context deadline exceeded; piecestore: (Node ID: 1tyoTVSF9udrfYpVx9fVmCgxN2T3qijzUVchzqXqbuq6rthBhs, Piece ID: HEPZDMAUXP7LWBCOTIKIU34UFWEIBCP4A7IEYDP7YKY4JJC3B3YA) context deadline exceeded; piecestore: (Node ID: 1ga5fKb8hWQFKK1MYdt8miZf8cfNPdk1kKUBCrrhuGPXPVpbDH, Piece ID: AAH7CGMFK3UXN6SSTDKIXZV6KLG3DTXQF72CN6RW6RAW4LXFNUMQ) context deadline exceeded; piecestore: (Node ID: 14zjhX7pamGsetyiEXPUmvjCFAxVVJeirWhbnABUySTHUCimGg, Piece ID: 3OP2J62KYOJISM64TF4GRRWNGFQPMR3VWNWUMJBW7QJKYPLLXSRQ) context deadline exceeded; piecestore: (Node ID: 12bcXTBAWidFQdAcAmTmPsHFrNb3KaicnKQoqtcnSyYPs8MfMR8, Piece ID: X2EKLRU6NCXKJWRCO6KBKGMAJWKAOYTWP6Y2HCANSD46WRNHBELQ) context deadline exceeded; piecestore: (Node ID: 12HtAGLF74ponLBttyfAin8MZV7oFBYvVCT1Jd5yKJG99anSVVC, Piece ID: GKFDL3UDEC2ML2DVSDCZVS3KQAEZ55AA7XGTVGO4PZ5XECCWPMZQ) context deadline exceeded; piecestore: (Node ID: 1Lbp5Ch18UPFNLALhockFsFSB7YPtbTmasHjFWcVXRVkPyp2q9, Piece ID: FTNIZAAK2HA6KOXWATPMEY6RI6CPKW4EUAQ4W2DA3AWSDE6K2LFA) context deadline exceeded Duration: 3.705688486s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.917594998s trace: https://jaeger.dev.storj.io/trace/2d0bbfdb45d7eb30 FAIL (2025-11-19 11:25:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Uploading piece... Nov 19 11:24:02.406153985 (243h20m0.957624446s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.406606356 (243h20m0.958076817s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407032668 (243h20m0.958503129s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407469961 (243h20m0.958940422s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.407868389 (243h20m0.95933885s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.408472380 (243h20m0.959942841s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409165069 (243h20m0.96063553s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.409636135 (243h20m0.961106596s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410111601 (243h20m0.961582062s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.410998800 (243h20m0.962469261s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.411605596 (243h20m0.963076057s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.412297504 (243h20m0.963767965s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413149025 (243h20m0.964619496s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.413855800 (243h20m0.965326261s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.414490900 (243h20m0.965961361s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:02.420844919 (243h20m0.97231538s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Uploading piece... Nov 19 11:24:03.904753662 (243h20m2.456224123s) [upload=214327]: Next returned nil segment Nov 19 11:24:03.904780122 (243h20m2.456250593s) [upload=214327]: Waiting for error group managing segments... Nov 19 11:24:04.299267328 (243h20m2.850737799s) [noise=true , address=145.79.227.18:4009 , piece_id=TU55EWKZIKI2O52HDC42K52LF4OPGGS5YZQUNFG7BVW2LYWQF7QQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.333727086 (243h20m2.885197547s) [noise=true , address=145.79.227.49:4356 , piece_id=LW4O5WR7QWY4DMIEHZHGHMM4J3MDVHSJSQUWZ2AQ2DZ4EY5QOK5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.360542317 (243h20m2.912012778s) [noise=true , address=145.79.227.36:4544 , piece_id=IVA4XSERVFRYMZX54DGSOPDBBFOJK6YPLHXDP7TW5QYQFLIEGBOQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.367107077 (243h20m2.918577548s) [noise=true , address=145.79.227.35:4515 , piece_id=QWJNZZ2ENTMV4X4V3JPWCP5DEC674T6MXJGHJ5NRXCOSAJVQUNJA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.403798734 (243h20m2.955269195s) [noise=true , address=145.79.227.13:4824 , piece_id=AP3MOXFUNQMOTTG3JHCCSOTBNAK7AEQ2SG7NXGRJ5H4C6QTR3TFA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:04.894325116 (243h20m3.445795577s) [noise=true , address=45.91.116.6:4999 , piece_id=L2K675IKROOJ5L2TAPJSKLTUCAWODHAXZY2BSOYQGZDOZ6PGQMMQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:05.844384733 (243h20m4.395855204s) [noise=true , address=45.91.116.8:4053 , piece_id=IDUTC52KSPSX2ZQPQ7NXZJWNDIEOLN6CXWYDLNZA4HMYNBYSXKSQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:06.237343548 (243h20m4.788813999s) [noise=true , address=77.90.31.21:4880 , piece_id=RPTXIGMRI24BQJVULSLUPXGSEQ6GPVG7C47WQ55G4R2JSG2Z5UAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.487081407 (243h20m6.038551878s) [noise=true , address=45.91.116.7:4014 , piece_id=WY5YXACDFD3IZADY4I5M43L75OIGKJPOWPKEFYMQZ66XVOJV4TVA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:07.785481827 (243h20m6.336952288s) [noise=true , address=77.90.31.15:4730 , piece_id=3TZGN5Y3XHRZEYVQVPKN3QWUG3E7JWRV2XV5R6AU57UNEVYBP6YQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:11.053363289 (243h20m9.60483375s) [noise=true , address=77.90.31.11:4587 , piece_id=Q6HOJMRMHBCSP2HLFNAU5ROBDZJMGCCSP43S3LUZZGE3NRCHKAAQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:16.699756801 (243h20m15.251227272s) [noise=true , address=77.90.31.17:4784 , piece_id=GUWJINOM3UOVQ72KO6JBS5KUYIRMAHIQUKECUCGNI4RN32KXZ5SA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:18.065310225 (243h20m16.616780686s) [noise=true , address=77.90.31.14:4675 , piece_id=PTN3DMHN5EGM4ZJVPF3MDFADHUSSP3MBP5NQRMVGGBGGZXDLZQPQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093276468 (243h20m30.64474696s) [noise=true , address=77.90.31.16:4759 , piece_id=542TUTLHICKT3AVLR2C5FZWVWXRPE4YRNYWN5HGJZ4ADY3CZNNDA, seg_pos={0 0}, upload=214327]: Done uploading piece. err: <nil> Nov 19 11:24:32.093311826 (243h20m30.644782307s) [seg_pos={0 0}, upload=214327]: Segment reached optimal threshold of 14 pieces. Nov 19 11:24:32.093740543 (243h20m30.645211044s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xWshtZa3oDwTV9555JzvCGvaEz6asXBPUuwobeoe3vgcwN84B): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.093794385 (243h20m30.645264846s) [noise=true , address=45.91.116.10:4095 , piece_id=FYU6VFXLEU5FKPWAM6RUCSQTX62PEU53P774GJXNN3QQL64JRD3Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.093850041 (243h20m30.645320562s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: read tcp 188.166.225.170:51688->45.91.116.5:4324: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.093860942 (243h20m30.645331403s) [noise=true , address=45.91.116.5:4324 , piece_id=VEGG7KYYNWV2RAGWHYX3TBH7PTAZDCROFBXMICO3Z2YERYSNYSKQ, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094054120 (243h20m30.645524591s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YGD2pRZaRuK1njBRQsrrrxHoLsK5TjhsckYjc3yTKjTFWHz3V): protocol: expected piece hash; context canceled; EOF Nov 19 11:24:32.094068026 (243h20m30.645538487s) [noise=true , address=45.91.116.9:4291 , piece_id=K36PRA2BAF6MOMQ46SHWZGOCKRMHCCF4JRP2INCBRKD4OSOYIK5Q, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094365111 (243h20m30.645835572s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FCikLXgPYS2qM9zLTthSNHKsQNckzg9oNkKGxfvwJZZjhKkVf): protocol: read tcp 188.166.225.170:49332->77.90.31.18:4814: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:32.094375041 (243h20m30.645845502s) [noise=true , address=77.90.31.18:4814 , piece_id=HIAO4VLMEXDWP2W3JK7J6YFM7WTAO5VNBH4LBJX5DTHVP7F62F5A, seg_pos={0 0}, upload=214327]: Long tail cancellation due to optimal threshold Nov 19 11:24:32.094397894 (243h20m30.645868355s) [seg_pos={0 0}, upload=214327]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Nov 19 11:24:32.094488376 (243h20m30.645958847s) [upload=214327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 19 11:24:32.094495089 (243h20m30.64596556s) [upload=214327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 19 11:24:32.413715534 (243h20m30.965185985s) [upload=214327]: Done uploading segments... Nov 19 11:24:32.413737264 (243h20m30.965207725s) [upload=214327]: Upload finished. err: <nil> Duration: 5.917624986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.954030127s trace: https://jaeger.dev.storj.io/trace/54d7555c2a2ae5e6 FAIL (2025-11-19 11:25:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 170707 (243h19m57.402641168s) [upload=214324]: Starting upload Nov 19 11:23:58.851267412 (243h19m57.402737873s) [upload=214324]: Done starting upload Nov 19 11:23:58.851359157 (243h19m57.402829618s) [upload=214324]: Uploading segments... Nov 19 11:23:58.851533098 (243h19m57.403003549s) [upload=214324]: Got next segment. Inline: false Nov 19 11:23:58.851579085 (243h19m57.403049546s) [upload=214324]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 19 11:23:58.851586489 (243h19m57.40305695s) [upload=214324]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 19 11:24:09.346928285 (243h20m7.898398746s) [seg_pos={0 0}, upload=214324]: Begin upload segment... Nov 19 11:24:09.346971878 (243h20m7.898442329s) [seg_pos={0 0}, upload=214324]: Done begin upload segment. Nov 19 11:24:09.346997186 (243h20m7.898467647s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.347899535 (243h20m7.899369996s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.348751847 (243h20m7.900222338s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349253423 (243h20m7.900723884s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349722666 (243h20m7.901193117s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.349744538 (243h20m7.901214999s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350164648 (243h20m7.901635109s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350298463 (243h20m7.901768924s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.350985580 (243h20m7.902456041s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.351646960 (243h20m7.903117421s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352106045 (243h20m7.903576496s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352226704 (243h20m7.903697165s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352502589 (243h20m7.903973051s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.352808733 (243h20m7.904279194s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353355172 (243h20m7.904825633s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:09.353960826 (243h20m7.905431287s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Uploading piece... Nov 19 11:24:11.079870614 (243h20m9.631341075s) [upload=214324]: Next returned nil segment Nov 19 11:24:11.079888597 (243h20m9.631359088s) [upload=214324]: Waiting for error group managing segments... Nov 19 11:24:11.370368685 (243h20m9.921839136s) [noise=true , address=145.79.227.23:4546 , piece_id=QXF7RXRVDWVBX6JANYV5GNJ63SHJJS6B6Y2BPYZCBAAT4MWZQBCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.393957131 (243h20m9.945427582s) [noise=true , address=145.79.227.32:4836 , piece_id=C3PB7TPO5UBCAWLA3T6HXDED6QYFBI2RWLXCJYVP5O3ER57JCALQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.396900396 (243h20m9.948370857s) [noise=true , address=145.79.227.14:4837 , piece_id=A7S6LDPJW4D4YOBBHBQ2LPCTRDV7GFS5PGHIP66PAATQMGLSDJAQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.406555350 (243h20m9.958025811s) [noise=true , address=145.79.227.8:4330 , piece_id=GAFAWMMGASFCFNJXTO4IV4AUFRXKCZXGCHW7FKNXQET2RNQ2KEDQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.408203239 (243h20m9.9596737s) [noise=true , address=145.79.227.28:4726 , piece_id=VPY57YYX3AIUDGJOBYUYSGEC7IBCOJPJGOCS5SBDC6Y4ABD5B6JA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.427732886 (243h20m9.979203348s) [noise=true , address=145.79.227.25:4618 , piece_id=TU7M5ZK6A7ZRZONBSNK3VDMHLGNB4LARMZOQSSSNKZ5AOWNMZHOQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.434574545 (243h20m9.986045006s) [noise=true , address=145.79.227.31:4797 , piece_id=4WSJOYT5PUJRVDRTVWO4TQ6UE5HVEETY2HHB7QITWWT424T6VW6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:11.965623448 (243h20m10.517093909s) [noise=true , address=45.91.116.6:4973 , piece_id=DLZMOXUEJVXSVCLKRUK2U65MWVW73OJAX4X233ACRCPMZMM36EVQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: <nil> Nov 19 11:24:12.704244297 (243h20m11.255714758s) [noise=true , address=45.91.116.13:4154 , piece_id=3H63BGHT2WK734HWVE3RINT3BLLZB6KJYD7O4EAYFHNSFNPMT2IQ, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704310923 (243h20m11.255781395s) [noise=true , address=77.90.31.17:4770 , piece_id=XBAPK4DJH5LTPMMCDUKDF32EMS46MCHDGW5NIOJVJ26PWCQVVMMA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704530301 (243h20m11.256000762s) [noise=true , address=45.91.116.11:4119 , piece_id=6WCEXY2DXFSAY6YCG3KXBLR3WR33RDSQVMGCW3ZSQH77VJ5T7G5A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:46214->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704554738 (243h20m11.256025199s) [noise=true , address=77.90.31.13:4626 , piece_id=VMTG572TS7QO3QNDFWECEX2Z5W7GKPLJK7WQZJSJYOQATYUQD4OA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: expected piece hash; context deadline exceeded; EOF Nov 19 11:24:12.704718500 (243h20m11.256188951s) [noise=true , address=45.91.116.14:4191 , piece_id=FPXHDVQGASXOT4P33BD3BZHWIAYZGJXCJ53AFVFBTGHDWPP6SKGA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: read tcp 188.166.225.170:59364->45.91.116.14:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.704860580 (243h20m11.256331041s) [noise=true , address=45.91.116.12:4225 , piece_id=H6D4TZI5TGNSEJOJYCBQWC6XQ6DLDJOBSET2I72QWE2BAFZ7XYCA, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, address:45.91.116.12:4225): protocol: read tcp 188.166.225.170:35728->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705105717 (243h20m11.256576178s) [noise=true , address=45.91.116.7:4009 , piece_id=VXQYDDRZX56SG6ALDXZHSYDMS5KR2IRFRQI4X64KF6CEOEFLMU6Q, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:33754->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705230465 (243h20m11.256700926s) [noise=true , address=45.91.116.10:4082 , piece_id=3CDP62UO2AOLDQJE6LQRRVVQTLFNQWJYLSAX5ETOBSXA7XHWNQ7A, seg_pos={0 0}, upload=214324]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: read tcp 188.166.225.170:53058->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 19 11:24:12.705281462 (243h20m11.256751923s) [seg_pos={0 0}, upload=214324]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 19 11:24:13.954848295 (243h20m12.506318746s) [upload=214324]: Done uploading segments... Nov 19 11:24:13.954922786 (243h20m12.506393257s) [upload=214324]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.954058851s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.880768695s trace: https://jaeger.dev.storj.io/trace/4a8bdfd70deab53a FAIL (2025-11-19 11:24:49 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.880888864s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)