Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 903790 reports (Page 11286 of 18076)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001301111s trace: https://jaeger.dev.storj.io/trace/4f2aeab421460ab2 FAIL (2025-12-29 14:22:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001341439s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.XXXXX..XX.X] duration: 30.002284173s trace: https://jaeger.dev.storj.io/trace/4d0abf88f725b9d FAIL (2025-12-29 14:21:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 2Y3GC25NZ6NXZYKXWJSJPX6VETGFLJMRYYNNHGW7XKEWRNTIEBEA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: HWNPUU5PWVPWFGBXY4GTZPU4MDNS3KTDABTO2UQXN4B57JANTKAA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: VDXS5SCXV2EL63HP3ITGNPBBRJZ53CWVFOWETDICPBVTULNNM7NA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: TSKHTPZCBZVDUYMZQRLZCZM3KYUL6M3P5ZASMYWOTNYEFKEX4PPQ) context deadline exceeded ======LOGS====== Dec 29 14:21:28.469717557 (574h52m28.340800592s) [upload=503967]: Starting upload Dec 29 14:21:28.469967462 (574h52m28.341050527s) [upload=503967]: Done starting upload Dec 29 14:21:28.470061300 (574h52m28.341144335s) [upload=503967]: Uploading segments... Dec 29 14:21:28.470218881 (574h52m28.341301916s) [upload=503967]: Got next segment. Inline: false Dec 29 14:21:28.470251673 (574h52m28.341334708s) [upload=503967]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:21:28.470286029 (574h52m28.341369064s) [upload=503967]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:21:28.720236441 (574h52m28.591319476s) [seg_pos={0 0}, upload=503967]: Begin upload segment... Dec 29 14:21:28.720286596 (574h52m28.591369631s) [seg_pos={0 0}, upload=503967]: Done begin upload segment. Dec 29 14:21:28.720330961 (574h52m28.591413996s) [noise=true , address=45.91.116.9:4264 , piece_id=HWNPUU5PWVPWFGBXY4GTZPU4MDNS3KTDABTO2UQXN4B57JANTKAA, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.720976891 (574h52m28.592059926s) [noise=true , address=145.79.227.7:4294 , piece_id=EKEO33JBJSNCRJ6WNWKTNNEG3TXACQ2Z6G7424TDTW5JHW3S7XWQ, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.722115880 (574h52m28.593198915s) [noise=true , address=77.90.31.12:4589 , piece_id=VDXS5SCXV2EL63HP3ITGNPBBRJZ53CWVFOWETDICPBVTULNNM7NA, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.722407815 (574h52m28.59349085s) [noise=true , address=45.91.116.5:4299 , piece_id=LTU54J74FGG2IOZTYIWV64ULL64TXYU5HEL536TDJD6L4TR3W6WA, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.722665046 (574h52m28.593748081s) [noise=true , address=145.79.227.45:4230 , piece_id=NV7LZBU5KFXMLIR2WG7STQMKWVVSCGM5ORF6OYPBEJCA4KEF7XKA, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.722964035 (574h52m28.59404707s) [noise=true , address=145.79.227.41:4085 , piece_id=7ZGDLVGSWFZHQ67KJHAUKI7WAY2LI4XG2TG57QVZAOEGNBCYLCLQ, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.723216435 (574h52m28.59429947s) [noise=true , address=145.79.227.30:4761 , piece_id=FVXE5QS5DRSZEC7VVQVQ2CN6JWMFPEWTHH725NY5FBYKTSBUTCQQ, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.723487161 (574h52m28.594570196s) [noise=true , address=145.79.227.48:4302 , piece_id=67IMGE2GXEHGQRYWBG76CUYNZR3IESZ2FUTH7RKBZJAFAOW6FNXQ, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.723752757 (574h52m28.594835792s) [noise=true , address=45.91.116.6:4974 , piece_id=TSKHTPZCBZVDUYMZQRLZCZM3KYUL6M3P5ZASMYWOTNYEFKEX4PPQ, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.724026448 (574h52m28.595109493s) [noise=true , address=45.91.116.11:4118 , piece_id=ER6FQYOIEUT5OGKU5D3XHMVNQNZZIVXBVWNAPIIAFERDUALQ5OMA, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.725680968 (574h52m28.596764013s) [noise=true , address=145.79.227.39:4015 , piece_id=YHVPVFHDC4DHT3QPJHAVOM3MYJMBUW66AVCFZXQAABLVSKNNGAQA, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.726104074 (574h52m28.597187119s) [noise=true , address=77.90.31.10:4517 , piece_id=FM3OQYQS7RDLM6M4AHCJ4YBWSCZTWRJX67QSXD36FBXCQX7DDQYQ, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.726399296 (574h52m28.597482331s) [noise=true , address=77.90.31.14:4664 , piece_id=FKFICKAMCWJMBKNILIDWI24WDZ4WLLEZL7YVOPVCIIMS6MROF7ZA, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.726694959 (574h52m28.597777994s) [noise=true , address=145.79.227.29:4941 , piece_id=NJIP7MHCQRVPM6VIIZLHG7CDVPS7VHJWRQ36MRPVSZX4VR3ZQVZQ, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.726972868 (574h52m28.598055903s) [noise=true , address=77.90.31.13:4629 , piece_id=77GZZTWRKUN4HC7VFXW4JHYNQULVX7GXPA2IQK66D2ZHTO2RDEAA, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:28.727282508 (574h52m28.598365543s) [noise=true , address=45.91.116.10:4083 , piece_id=2Y3GC25NZ6NXZYKXWJSJPX6VETGFLJMRYYNNHGW7XKEWRNTIEBEA, seg_pos={0 0}, upload=503967]: Uploading piece... Dec 29 14:21:30.418175168 (574h52m30.289258203s) [upload=503967]: Next returned nil segment Dec 29 14:21:30.418187752 (574h52m30.289270787s) [upload=503967]: Waiting for error group managing segments... Dec 29 14:21:30.705492914 (574h52m30.576575949s) [noise=true , address=145.79.227.41:4085 , piece_id=7ZGDLVGSWFZHQ67KJHAUKI7WAY2LI4XG2TG57QVZAOEGNBCYLCLQ, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:30.717459511 (574h52m30.588542546s) [noise=true , address=145.79.227.7:4294 , piece_id=EKEO33JBJSNCRJ6WNWKTNNEG3TXACQ2Z6G7424TDTW5JHW3S7XWQ, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:30.723831915 (574h52m30.59491496s) [noise=true , address=145.79.227.39:4015 , piece_id=YHVPVFHDC4DHT3QPJHAVOM3MYJMBUW66AVCFZXQAABLVSKNNGAQA, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:30.734376855 (574h52m30.60545989s) [noise=true , address=145.79.227.48:4302 , piece_id=67IMGE2GXEHGQRYWBG76CUYNZR3IESZ2FUTH7RKBZJAFAOW6FNXQ, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:30.744086232 (574h52m30.615169267s) [noise=true , address=145.79.227.45:4230 , piece_id=NV7LZBU5KFXMLIR2WG7STQMKWVVSCGM5ORF6OYPBEJCA4KEF7XKA, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:30.889587097 (574h52m30.760670132s) [noise=true , address=77.90.31.12:4589 , piece_id=VDXS5SCXV2EL63HP3ITGNPBBRJZ53CWVFOWETDICPBVTULNNM7NA, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:30.919581922 (574h52m30.790664957s) [noise=true , address=77.90.31.10:4517 , piece_id=FM3OQYQS7RDLM6M4AHCJ4YBWSCZTWRJX67QSXD36FBXCQX7DDQYQ, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:30.963414295 (574h52m30.83449733s) [noise=true , address=45.91.116.11:4118 , piece_id=ER6FQYOIEUT5OGKU5D3XHMVNQNZZIVXBVWNAPIIAFERDUALQ5OMA, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:31.051603521 (574h52m30.922686566s) [noise=true , address=45.91.116.10:4083 , piece_id=2Y3GC25NZ6NXZYKXWJSJPX6VETGFLJMRYYNNHGW7XKEWRNTIEBEA, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:31.096553542 (574h52m30.967636577s) [noise=true , address=77.90.31.14:4664 , piece_id=FKFICKAMCWJMBKNILIDWI24WDZ4WLLEZL7YVOPVCIIMS6MROF7ZA, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:31.098537300 (574h52m30.969620335s) [noise=true , address=45.91.116.6:4974 , piece_id=TSKHTPZCBZVDUYMZQRLZCZM3KYUL6M3P5ZASMYWOTNYEFKEX4PPQ, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:31.284671847 (574h52m31.155754892s) [noise=true , address=45.91.116.9:4264 , piece_id=HWNPUU5PWVPWFGBXY4GTZPU4MDNS3KTDABTO2UQXN4B57JANTKAA, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:31.410347762 (574h52m31.281430797s) [noise=true , address=77.90.31.13:4629 , piece_id=77GZZTWRKUN4HC7VFXW4JHYNQULVX7GXPA2IQK66D2ZHTO2RDEAA, seg_pos={0 0}, upload=503967]: Done uploading piece. err: <nil> Dec 29 14:21:31.410374273 (574h52m31.281457308s) [seg_pos={0 0}, upload=503967]: Segment reached optimal threshold of 13 pieces. Dec 29 14:21:31.410574864 (574h52m31.281657899s) [noise=true , address=145.79.227.29:4941 , piece_id=NJIP7MHCQRVPM6VIIZLHG7CDVPS7VHJWRQ36MRPVSZX4VR3ZQVZQ, seg_pos={0 0}, upload=503967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 29 14:21:31.410581257 (574h52m31.281664292s) [noise=true , address=145.79.227.29:4941 , piece_id=NJIP7MHCQRVPM6VIIZLHG7CDVPS7VHJWRQ36MRPVSZX4VR3ZQVZQ, seg_pos={0 0}, upload=503967]: Long tail cancellation due to optimal threshold Dec 29 14:21:31.410678081 (574h52m31.281761116s) [noise=true , address=145.79.227.30:4761 , piece_id=FVXE5QS5DRSZEC7VVQVQ2CN6JWMFPEWTHH725NY5FBYKTSBUTCQQ, seg_pos={0 0}, upload=503967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 14:21:31.410694312 (574h52m31.281777347s) [noise=true , address=145.79.227.30:4761 , piece_id=FVXE5QS5DRSZEC7VVQVQ2CN6JWMFPEWTHH725NY5FBYKTSBUTCQQ, seg_pos={0 0}, upload=503967]: Long tail cancellation due to optimal threshold Dec 29 14:21:31.410916004 (574h52m31.281999039s) [noise=true , address=45.91.116.5:4299 , piece_id=LTU54J74FGG2IOZTYIWV64ULL64TXYU5HEL536TDJD6L4TR3W6WA, seg_pos={0 0}, upload=503967]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: read tcp 188.166.225.170:50526->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 14:21:31.410922486 (574h52m31.282005521s) [noise=true , address=45.91.116.5:4299 , piece_id=LTU54J74FGG2IOZTYIWV64ULL64TXYU5HEL536TDJD6L4TR3W6WA, seg_pos={0 0}, upload=503967]: Long tail cancellation due to optimal threshold Dec 29 14:21:31.410938788 (574h52m31.282021833s) [seg_pos={0 0}, upload=503967]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:21:31.411040672 (574h52m31.282123707s) [upload=503967]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:21:31.411048206 (574h52m31.282131251s) [upload=503967]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:21:31.703069116 (574h52m31.574152152s) [upload=503967]: Done uploading segments... Dec 29 14:21:31.703089456 (574h52m31.574172501s) [upload=503967]: Upload finished. err: <nil> Duration: 30.002301797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................XXX..X..XX.....................] duration: 55.793027713s trace: https://jaeger.dev.storj.io/trace/266137264da4c90 FAIL (2025-12-29 14:21:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded ======LOGS====== NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments... Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces. Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments... Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil> Duration: 55.793043513s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.....X...X.X.XX....X........X.X............X..X.XX.XXX.X.X..] duration: 19.348075134s trace: https://jaeger.dev.storj.io/trace/6e70f23aaeb95654 FAIL (2025-12-29 14:21:54 UTC) orbiter warning: context deadline exceeded Duration: 19.348089553s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...X...........X...X....X....X.XX...........X...............] duration: 16.695646056s trace: https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd FAIL (2025-12-29 14:21:49 UTC) orbiter warning: context deadline exceeded Duration: 16.695672866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001716604s trace: https://jaeger.dev.storj.io/trace/347f3f2b1d5aeb33 FAIL (2025-12-29 14:21:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00177717s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [......X...X.X.XX....X........X.X............X..X.XX.XXX.X.X.] duration: 25.70914158s trace: https://jaeger.dev.storj.io/trace/6e70f23aaeb95654 FAIL (2025-12-29 14:21:34 UTC) orbiter warning: context deadline exceeded Duration: 25.709170806s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000626516s trace: https://jaeger.dev.storj.io/trace/1e81c0f0af486405 FAIL (2025-12-29 14:21:24 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000649741s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......X...X.X.XX....X........X.X............X..X.XX.XXX.X.X] duration: 30.001093076s trace: https://jaeger.dev.storj.io/trace/6e70f23aaeb95654 FAIL (2025-12-29 14:21:07 UTC) orbiter warning: context deadline exceeded Duration: 30.00111677s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001619148s trace: https://jaeger.dev.storj.io/trace/4a87d1d7eac5cbf8 FAIL (2025-12-29 14:21:06 UTC) orbiter warning: context deadline exceeded Duration: 5.001642853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [....X...........X...X....X....X.XX...........X..............] duration: 28.077538982s trace: https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd FAIL (2025-12-29 14:21:05 UTC) orbiter warning: context deadline exceeded Duration: 28.077565411s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................XXX..X..XX....................] duration: 59.661915533s trace: https://jaeger.dev.storj.io/trace/266137264da4c90 FAIL (2025-12-29 14:20:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded ======LOGS====== NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments... Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces. Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments... Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil> Duration: 59.66194026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.XXXXX..XX.] duration: 6.708088229s trace: https://jaeger.dev.storj.io/trace/1396ee0c89510e2e FAIL (2025-12-29 14:20:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: UJUDMBVOIK56ONZSAZA4T776NJOJFWJB3ZSHASWR665FSYIE3NAA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: KLCFPEZ2KGIGHTVTNSPQD3PK6DF2R6HFE7X45A4RGQSVRUX3YUKQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: AOYAOTDQOJKWLD4XE2XCR67YGOA6LG5DUSJOHSUS6Q5HAQJKJQGA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 46KFXX7VKIP35LDDTLKACTIZWQ4ZD4XRMEG4MKNB376M3U6AZOXA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: GFDC3GIXJWUOXEOEG22LNK4BWXFRH72LQA42BUYVU7QFIGYZ5UMA) context deadline exceeded ======LOGS====== 29 14:19:45.383969885 (574h50m45.25505292s) [upload=503942]: Starting upload Dec 29 14:19:45.384033486 (574h50m45.255116521s) [upload=503942]: Done starting upload Dec 29 14:19:45.384209311 (574h50m45.255292356s) [upload=503942]: Uploading segments... Dec 29 14:19:45.384241161 (574h50m45.255324196s) [upload=503942]: Got next segment. Inline: false Dec 29 14:19:45.384265388 (574h50m45.255348423s) [upload=503942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:19:45.384271539 (574h50m45.255354584s) [upload=503942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:19:45.630699836 (574h50m45.501782881s) [seg_pos={0 0}, upload=503942]: Begin upload segment... Dec 29 14:19:45.630749991 (574h50m45.501833026s) [seg_pos={0 0}, upload=503942]: Done begin upload segment. Dec 29 14:19:45.630779337 (574h50m45.501862382s) [noise=true , address=145.79.227.35:4481 , piece_id=527H4QZUJEPPZHTBVUGTFDOKVIXXDOT4MY76SWZJ475CO5QOP4NA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.630848248 (574h50m45.501931283s) [noise=true , address=45.91.116.5:4297 , piece_id=WV2NINLXV5ZJMAJHQYI3BIBMPO4YHHDERAH7YJ6WHF6TCYEQUFNQ, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.631280140 (574h50m45.502363175s) [noise=true , address=145.79.227.39:4015 , piece_id=WQ45EEKH6JFRFSPJJDXRRB5RZGZCXYCBOP4HNHYKDLDTL3GNEY6Q, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.631570173 (574h50m45.502653208s) [noise=true , address=45.91.116.6:4974 , piece_id=KLCFPEZ2KGIGHTVTNSPQD3PK6DF2R6HFE7X45A4RGQSVRUX3YUKQ, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.631575684 (574h50m45.502658719s) [noise=true , address=45.91.116.13:4156 , piece_id=AOYAOTDQOJKWLD4XE2XCR67YGOA6LG5DUSJOHSUS6Q5HAQJKJQGA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.631837462 (574h50m45.502920507s) [noise=true , address=45.91.116.12:4228 , piece_id=ETDFDDQT2CNZMTST34BPUL464YS67WAWXDFKBF6IS3YN2JVM7WAA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.631974052 (574h50m45.503057097s) [noise=true , address=45.91.116.10:4081 , piece_id=46KFXX7VKIP35LDDTLKACTIZWQ4ZD4XRMEG4MKNB376M3U6AZOXA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632117214 (574h50m45.503200259s) [noise=true , address=45.91.116.7:4012 , piece_id=3FOMZJQNG6HUETF5AWI2BBOPNZ23U5BIEVMP6OOVKZRNTPN4V2OQ, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632269355 (574h50m45.50335239s) [noise=true , address=145.79.227.41:4086 , piece_id=NUTMSGJBNYIFFS6UKZMCFM7BB6M4PLSO2JVVNCRI3EH7P3JMGB6A, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632388190 (574h50m45.503471226s) [noise=true , address=145.79.227.52:4445 , piece_id=EM6S2T2ORYQD7PLHGA4ACO5OM3UDYVZIKBTWGZCYRPF2WCQ6NZAA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632544649 (574h50m45.503627684s) [noise=true , address=145.79.227.5:4223 , piece_id=FZMLKSTR5ZJ3KJHRREZJ3CEWOEF3DIRVO5FQUHQVFXXRIK24A7NA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632655850 (574h50m45.503738886s) [noise=true , address=45.91.116.11:4119 , piece_id=4S2GOBMH3GNM3D6O27FA3HV4SENTCDADY5FAKHX5DRN2HYR5OVSA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632844870 (574h50m45.503927905s) [noise=true , address=77.90.31.12:4590 , piece_id=GFDC3GIXJWUOXEOEG22LNK4BWXFRH72LQA42BUYVU7QFIGYZ5UMA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632950922 (574h50m45.504033957s) [noise=true , address=145.79.227.17:4970 , piece_id=3NCEURD6EAROKYW3TERQHJGPK3EEHIFDUEQGKDQMHAVUW7AU6JFQ, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.633118902 (574h50m45.504201947s) [noise=true , address=45.91.116.9:4264 , piece_id=UJUDMBVOIK56ONZSAZA4T776NJOJFWJB3ZSHASWR665FSYIE3NAA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.633389437 (574h50m45.504472472s) [noise=true , address=45.91.116.14:4190 , piece_id=J5XLORJQYYY27QJWW4IDKUKXUBGVKPKHIJ3OTRMDFWVZLRX52AIA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:47.326312867 (574h50m47.197395902s) [upload=503942]: Next returned nil segment Dec 29 14:19:47.326328366 (574h50m47.197411411s) [upload=503942]: Waiting for error group managing segments... Dec 29 14:19:47.617316480 (574h50m47.488399535s) [noise=true , address=145.79.227.39:4015 , piece_id=WQ45EEKH6JFRFSPJJDXRRB5RZGZCXYCBOP4HNHYKDLDTL3GNEY6Q, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.620998301 (574h50m47.492081336s) [noise=true , address=145.79.227.41:4086 , piece_id=NUTMSGJBNYIFFS6UKZMCFM7BB6M4PLSO2JVVNCRI3EH7P3JMGB6A, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.622005489 (574h50m47.493088524s) [noise=true , address=145.79.227.17:4970 , piece_id=3NCEURD6EAROKYW3TERQHJGPK3EEHIFDUEQGKDQMHAVUW7AU6JFQ, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.638944894 (574h50m47.510027939s) [noise=true , address=145.79.227.5:4223 , piece_id=FZMLKSTR5ZJ3KJHRREZJ3CEWOEF3DIRVO5FQUHQVFXXRIK24A7NA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.664270838 (574h50m47.535353874s) [noise=true , address=145.79.227.52:4445 , piece_id=EM6S2T2ORYQD7PLHGA4ACO5OM3UDYVZIKBTWGZCYRPF2WCQ6NZAA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.861397246 (574h50m47.732480281s) [noise=true , address=45.91.116.5:4297 , piece_id=WV2NINLXV5ZJMAJHQYI3BIBMPO4YHHDERAH7YJ6WHF6TCYEQUFNQ, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.893961845 (574h50m47.76504489s) [noise=true , address=45.91.116.10:4081 , piece_id=46KFXX7VKIP35LDDTLKACTIZWQ4ZD4XRMEG4MKNB376M3U6AZOXA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.904524658 (574h50m47.775607693s) [noise=true , address=45.91.116.13:4156 , piece_id=AOYAOTDQOJKWLD4XE2XCR67YGOA6LG5DUSJOHSUS6Q5HAQJKJQGA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.990517851 (574h50m47.861600896s) [noise=true , address=45.91.116.7:4012 , piece_id=3FOMZJQNG6HUETF5AWI2BBOPNZ23U5BIEVMP6OOVKZRNTPN4V2OQ, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:48.025755359 (574h50m47.896838394s) [noise=true , address=77.90.31.12:4590 , piece_id=GFDC3GIXJWUOXEOEG22LNK4BWXFRH72LQA42BUYVU7QFIGYZ5UMA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:48.080818998 (574h50m47.951902033s) [noise=true , address=45.91.116.12:4228 , piece_id=ETDFDDQT2CNZMTST34BPUL464YS67WAWXDFKBF6IS3YN2JVM7WAA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:48.104458552 (574h50m47.975541597s) [noise=true , address=45.91.116.9:4264 , piece_id=UJUDMBVOIK56ONZSAZA4T776NJOJFWJB3ZSHASWR665FSYIE3NAA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:48.604988134 (574h50m48.476071169s) [noise=true , address=45.91.116.6:4974 , piece_id=KLCFPEZ2KGIGHTVTNSPQD3PK6DF2R6HFE7X45A4RGQSVRUX3YUKQ, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:48.605013391 (574h50m48.476096476s) [seg_pos={0 0}, upload=503942]: Segment reached optimal threshold of 13 pieces. Dec 29 14:19:48.605273126 (574h50m48.476356171s) [noise=true , address=45.91.116.11:4119 , piece_id=4S2GOBMH3GNM3D6O27FA3HV4SENTCDADY5FAKHX5DRN2HYR5OVSA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 14:19:48.605281893 (574h50m48.476364928s) [noise=true , address=45.91.116.11:4119 , piece_id=4S2GOBMH3GNM3D6O27FA3HV4SENTCDADY5FAKHX5DRN2HYR5OVSA, seg_pos={0 0}, upload=503942]: Long tail cancellation due to optimal threshold Dec 29 14:19:48.605344342 (574h50m48.476427377s) [noise=true , address=145.79.227.35:4481 , piece_id=527H4QZUJEPPZHTBVUGTFDOKVIXXDOT4MY76SWZJ475CO5QOP4NA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 14:19:48.605368027 (574h50m48.476451072s) [noise=true , address=145.79.227.35:4481 , piece_id=527H4QZUJEPPZHTBVUGTFDOKVIXXDOT4MY76SWZJ475CO5QOP4NA, seg_pos={0 0}, upload=503942]: Long tail cancellation due to optimal threshold Dec 29 14:19:48.605534964 (574h50m48.476618s) [noise=true , address=45.91.116.14:4190 , piece_id=J5XLORJQYYY27QJWW4IDKUKXUBGVKPKHIJ3OTRMDFWVZLRX52AIA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:39384->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:39384->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39384->45.91.116.14:4190: use of closed network connection Dec 29 14:19:48.605540996 (574h50m48.476624031s) [noise=true , address=45.91.116.14:4190 , piece_id=J5XLORJQYYY27QJWW4IDKUKXUBGVKPKHIJ3OTRMDFWVZLRX52AIA, seg_pos={0 0}, upload=503942]: Long tail cancellation due to optimal threshold Dec 29 14:19:48.605560974 (574h50m48.476644009s) [seg_pos={0 0}, upload=503942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:19:48.605637169 (574h50m48.476720214s) [upload=503942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:19:48.605641617 (574h50m48.476724653s) [upload=503942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:19:48.892187634 (574h50m48.763270679s) [upload=503942]: Done uploading segments... Dec 29 14:19:48.892206460 (574h50m48.763289505s) [upload=503942]: Upload finished. err: <nil> Duration: 6.708112916s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001218013s trace: https://jaeger.dev.storj.io/trace/745670bf2b89ef57 FAIL (2025-12-29 14:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001241388s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.XXXXX..XX] duration: 30.002504475s trace: https://jaeger.dev.storj.io/trace/1396ee0c89510e2e FAIL (2025-12-29 14:20:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: UJUDMBVOIK56ONZSAZA4T776NJOJFWJB3ZSHASWR665FSYIE3NAA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: KLCFPEZ2KGIGHTVTNSPQD3PK6DF2R6HFE7X45A4RGQSVRUX3YUKQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: AOYAOTDQOJKWLD4XE2XCR67YGOA6LG5DUSJOHSUS6Q5HAQJKJQGA) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 46KFXX7VKIP35LDDTLKACTIZWQ4ZD4XRMEG4MKNB376M3U6AZOXA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: GFDC3GIXJWUOXEOEG22LNK4BWXFRH72LQA42BUYVU7QFIGYZ5UMA) context deadline exceeded ======LOGS====== 29 14:19:45.383969885 (574h50m45.25505292s) [upload=503942]: Starting upload Dec 29 14:19:45.384033486 (574h50m45.255116521s) [upload=503942]: Done starting upload Dec 29 14:19:45.384209311 (574h50m45.255292356s) [upload=503942]: Uploading segments... Dec 29 14:19:45.384241161 (574h50m45.255324196s) [upload=503942]: Got next segment. Inline: false Dec 29 14:19:45.384265388 (574h50m45.255348423s) [upload=503942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:19:45.384271539 (574h50m45.255354584s) [upload=503942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:19:45.630699836 (574h50m45.501782881s) [seg_pos={0 0}, upload=503942]: Begin upload segment... Dec 29 14:19:45.630749991 (574h50m45.501833026s) [seg_pos={0 0}, upload=503942]: Done begin upload segment. Dec 29 14:19:45.630779337 (574h50m45.501862382s) [noise=true , address=145.79.227.35:4481 , piece_id=527H4QZUJEPPZHTBVUGTFDOKVIXXDOT4MY76SWZJ475CO5QOP4NA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.630848248 (574h50m45.501931283s) [noise=true , address=45.91.116.5:4297 , piece_id=WV2NINLXV5ZJMAJHQYI3BIBMPO4YHHDERAH7YJ6WHF6TCYEQUFNQ, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.631280140 (574h50m45.502363175s) [noise=true , address=145.79.227.39:4015 , piece_id=WQ45EEKH6JFRFSPJJDXRRB5RZGZCXYCBOP4HNHYKDLDTL3GNEY6Q, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.631570173 (574h50m45.502653208s) [noise=true , address=45.91.116.6:4974 , piece_id=KLCFPEZ2KGIGHTVTNSPQD3PK6DF2R6HFE7X45A4RGQSVRUX3YUKQ, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.631575684 (574h50m45.502658719s) [noise=true , address=45.91.116.13:4156 , piece_id=AOYAOTDQOJKWLD4XE2XCR67YGOA6LG5DUSJOHSUS6Q5HAQJKJQGA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.631837462 (574h50m45.502920507s) [noise=true , address=45.91.116.12:4228 , piece_id=ETDFDDQT2CNZMTST34BPUL464YS67WAWXDFKBF6IS3YN2JVM7WAA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.631974052 (574h50m45.503057097s) [noise=true , address=45.91.116.10:4081 , piece_id=46KFXX7VKIP35LDDTLKACTIZWQ4ZD4XRMEG4MKNB376M3U6AZOXA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632117214 (574h50m45.503200259s) [noise=true , address=45.91.116.7:4012 , piece_id=3FOMZJQNG6HUETF5AWI2BBOPNZ23U5BIEVMP6OOVKZRNTPN4V2OQ, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632269355 (574h50m45.50335239s) [noise=true , address=145.79.227.41:4086 , piece_id=NUTMSGJBNYIFFS6UKZMCFM7BB6M4PLSO2JVVNCRI3EH7P3JMGB6A, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632388190 (574h50m45.503471226s) [noise=true , address=145.79.227.52:4445 , piece_id=EM6S2T2ORYQD7PLHGA4ACO5OM3UDYVZIKBTWGZCYRPF2WCQ6NZAA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632544649 (574h50m45.503627684s) [noise=true , address=145.79.227.5:4223 , piece_id=FZMLKSTR5ZJ3KJHRREZJ3CEWOEF3DIRVO5FQUHQVFXXRIK24A7NA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632655850 (574h50m45.503738886s) [noise=true , address=45.91.116.11:4119 , piece_id=4S2GOBMH3GNM3D6O27FA3HV4SENTCDADY5FAKHX5DRN2HYR5OVSA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632844870 (574h50m45.503927905s) [noise=true , address=77.90.31.12:4590 , piece_id=GFDC3GIXJWUOXEOEG22LNK4BWXFRH72LQA42BUYVU7QFIGYZ5UMA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.632950922 (574h50m45.504033957s) [noise=true , address=145.79.227.17:4970 , piece_id=3NCEURD6EAROKYW3TERQHJGPK3EEHIFDUEQGKDQMHAVUW7AU6JFQ, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.633118902 (574h50m45.504201947s) [noise=true , address=45.91.116.9:4264 , piece_id=UJUDMBVOIK56ONZSAZA4T776NJOJFWJB3ZSHASWR665FSYIE3NAA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:45.633389437 (574h50m45.504472472s) [noise=true , address=45.91.116.14:4190 , piece_id=J5XLORJQYYY27QJWW4IDKUKXUBGVKPKHIJ3OTRMDFWVZLRX52AIA, seg_pos={0 0}, upload=503942]: Uploading piece... Dec 29 14:19:47.326312867 (574h50m47.197395902s) [upload=503942]: Next returned nil segment Dec 29 14:19:47.326328366 (574h50m47.197411411s) [upload=503942]: Waiting for error group managing segments... Dec 29 14:19:47.617316480 (574h50m47.488399535s) [noise=true , address=145.79.227.39:4015 , piece_id=WQ45EEKH6JFRFSPJJDXRRB5RZGZCXYCBOP4HNHYKDLDTL3GNEY6Q, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.620998301 (574h50m47.492081336s) [noise=true , address=145.79.227.41:4086 , piece_id=NUTMSGJBNYIFFS6UKZMCFM7BB6M4PLSO2JVVNCRI3EH7P3JMGB6A, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.622005489 (574h50m47.493088524s) [noise=true , address=145.79.227.17:4970 , piece_id=3NCEURD6EAROKYW3TERQHJGPK3EEHIFDUEQGKDQMHAVUW7AU6JFQ, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.638944894 (574h50m47.510027939s) [noise=true , address=145.79.227.5:4223 , piece_id=FZMLKSTR5ZJ3KJHRREZJ3CEWOEF3DIRVO5FQUHQVFXXRIK24A7NA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.664270838 (574h50m47.535353874s) [noise=true , address=145.79.227.52:4445 , piece_id=EM6S2T2ORYQD7PLHGA4ACO5OM3UDYVZIKBTWGZCYRPF2WCQ6NZAA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.861397246 (574h50m47.732480281s) [noise=true , address=45.91.116.5:4297 , piece_id=WV2NINLXV5ZJMAJHQYI3BIBMPO4YHHDERAH7YJ6WHF6TCYEQUFNQ, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.893961845 (574h50m47.76504489s) [noise=true , address=45.91.116.10:4081 , piece_id=46KFXX7VKIP35LDDTLKACTIZWQ4ZD4XRMEG4MKNB376M3U6AZOXA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.904524658 (574h50m47.775607693s) [noise=true , address=45.91.116.13:4156 , piece_id=AOYAOTDQOJKWLD4XE2XCR67YGOA6LG5DUSJOHSUS6Q5HAQJKJQGA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:47.990517851 (574h50m47.861600896s) [noise=true , address=45.91.116.7:4012 , piece_id=3FOMZJQNG6HUETF5AWI2BBOPNZ23U5BIEVMP6OOVKZRNTPN4V2OQ, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:48.025755359 (574h50m47.896838394s) [noise=true , address=77.90.31.12:4590 , piece_id=GFDC3GIXJWUOXEOEG22LNK4BWXFRH72LQA42BUYVU7QFIGYZ5UMA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:48.080818998 (574h50m47.951902033s) [noise=true , address=45.91.116.12:4228 , piece_id=ETDFDDQT2CNZMTST34BPUL464YS67WAWXDFKBF6IS3YN2JVM7WAA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:48.104458552 (574h50m47.975541597s) [noise=true , address=45.91.116.9:4264 , piece_id=UJUDMBVOIK56ONZSAZA4T776NJOJFWJB3ZSHASWR665FSYIE3NAA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:48.604988134 (574h50m48.476071169s) [noise=true , address=45.91.116.6:4974 , piece_id=KLCFPEZ2KGIGHTVTNSPQD3PK6DF2R6HFE7X45A4RGQSVRUX3YUKQ, seg_pos={0 0}, upload=503942]: Done uploading piece. err: <nil> Dec 29 14:19:48.605013391 (574h50m48.476096476s) [seg_pos={0 0}, upload=503942]: Segment reached optimal threshold of 13 pieces. Dec 29 14:19:48.605273126 (574h50m48.476356171s) [noise=true , address=45.91.116.11:4119 , piece_id=4S2GOBMH3GNM3D6O27FA3HV4SENTCDADY5FAKHX5DRN2HYR5OVSA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 14:19:48.605281893 (574h50m48.476364928s) [noise=true , address=45.91.116.11:4119 , piece_id=4S2GOBMH3GNM3D6O27FA3HV4SENTCDADY5FAKHX5DRN2HYR5OVSA, seg_pos={0 0}, upload=503942]: Long tail cancellation due to optimal threshold Dec 29 14:19:48.605344342 (574h50m48.476427377s) [noise=true , address=145.79.227.35:4481 , piece_id=527H4QZUJEPPZHTBVUGTFDOKVIXXDOT4MY76SWZJ475CO5QOP4NA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 14:19:48.605368027 (574h50m48.476451072s) [noise=true , address=145.79.227.35:4481 , piece_id=527H4QZUJEPPZHTBVUGTFDOKVIXXDOT4MY76SWZJ475CO5QOP4NA, seg_pos={0 0}, upload=503942]: Long tail cancellation due to optimal threshold Dec 29 14:19:48.605534964 (574h50m48.476618s) [noise=true , address=45.91.116.14:4190 , piece_id=J5XLORJQYYY27QJWW4IDKUKXUBGVKPKHIJ3OTRMDFWVZLRX52AIA, seg_pos={0 0}, upload=503942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:39384->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:39384->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39384->45.91.116.14:4190: use of closed network connection Dec 29 14:19:48.605540996 (574h50m48.476624031s) [noise=true , address=45.91.116.14:4190 , piece_id=J5XLORJQYYY27QJWW4IDKUKXUBGVKPKHIJ3OTRMDFWVZLRX52AIA, seg_pos={0 0}, upload=503942]: Long tail cancellation due to optimal threshold Dec 29 14:19:48.605560974 (574h50m48.476644009s) [seg_pos={0 0}, upload=503942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:19:48.605637169 (574h50m48.476720214s) [upload=503942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:19:48.605641617 (574h50m48.476724653s) [upload=503942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:19:48.892187634 (574h50m48.763270679s) [upload=503942]: Done uploading segments... Dec 29 14:19:48.892206460 (574h50m48.763289505s) [upload=503942]: Upload finished. err: <nil> Duration: 30.002527889s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001180161s trace: https://jaeger.dev.storj.io/trace/155d03a54b73d1e4 FAIL (2025-12-29 14:20:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001224445s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........X...X.X.XX....X........X.X............X..X.XX.XXX.X.] duration: 20.485126935s trace: https://jaeger.dev.storj.io/trace/6afc00cfbd932405 FAIL (2025-12-29 14:20:02 UTC) orbiter warning: context deadline exceeded Duration: 20.485158466s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.....X...........X...X....X....X.XX...........X.............] duration: 18.421587638s trace: https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd FAIL (2025-12-29 14:19:56 UTC) orbiter warning: context deadline exceeded Duration: 18.421599711s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000973127s trace: https://jaeger.dev.storj.io/trace/7733e0e8948d4c0c FAIL (2025-12-29 14:19:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001037029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.XXXXX..X] duration: 30.002423385s trace: https://jaeger.dev.storj.io/trace/6269a772a6db1c63 FAIL (2025-12-29 14:19:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: Q7AV3TMIG4FOYUCFY3HOER5NOFGDNEN34PFD3PZ6EUIUQ2MZHMPA) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: QIZWLWBW7ISNMX3M63E7GY4VF2W7A5YL7MDHPUA4RUDJA5JNBZYA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: 2UM5EHD7LQ5Q4JDQQU25SSSZO3M42PA2ONFJCTSBV5YNPZUAMRWQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: LJQBVZEUV2ZCZ4GIFADV2OZAMPSZMSWIS4AKQ5DJWLYAX5B4SXQQ) context deadline exceeded ======LOGS====== Dec 29 14:19:11.318046022 (574h50m11.189129067s) [upload=503934]: Starting upload Dec 29 14:19:11.318136103 (574h50m11.189219138s) [upload=503934]: Done starting upload Dec 29 14:19:11.318247835 (574h50m11.18933089s) [upload=503934]: Uploading segments... Dec 29 14:19:11.318398944 (574h50m11.189481979s) [upload=503934]: Got next segment. Inline: false Dec 29 14:19:11.318448467 (574h50m11.189531533s) [upload=503934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:19:11.318454869 (574h50m11.189537915s) [upload=503934]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:19:11.564656354 (574h50m11.435739389s) [seg_pos={0 0}, upload=503934]: Begin upload segment... Dec 29 14:19:11.564730385 (574h50m11.43581342s) [seg_pos={0 0}, upload=503934]: Done begin upload segment. Dec 29 14:19:11.564779148 (574h50m11.435862404s) [noise=true , address=77.90.31.13:4627 , piece_id=V6ASI4XTUW2ECBNZVCG373SEWDXVQN2EWNHHTK5RDMEGLLNIMD2A, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.564800770 (574h50m11.435883805s) [noise=true , address=45.91.116.8:4048 , piece_id=LJQBVZEUV2ZCZ4GIFADV2OZAMPSZMSWIS4AKQ5DJWLYAX5B4SXQQ, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.565271786 (574h50m11.436354831s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ27XRRACEVZVUOC7MZLCFF4HBTB23BPMG7QPDH37J3VX7PXVQKQ, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.565592216 (574h50m11.436675261s) [noise=true , address=145.79.227.3:4150 , piece_id=LMIOVDMWV7Y5QITOLUF25DDVJB5WAO3VDFMDUHB3BQDV4NUOOTGA, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.566060348 (574h50m11.437143403s) [noise=true , address=145.79.227.16:4909 , piece_id=3GRC7HPRMYQCR55UPCKZVXAQACRVFNY43YTT5HW4TRFSGDWVPECQ, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.566397940 (574h50m11.437480975s) [noise=true , address=45.91.116.14:4189 , piece_id=Q7AV3TMIG4FOYUCFY3HOER5NOFGDNEN34PFD3PZ6EUIUQ2MZHMPA, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.566751755 (574h50m11.43783479s) [noise=true , address=45.91.116.12:4227 , piece_id=CM3L7NS5DYZM3XAACV7JKNNPHYWWGXECBLKUPTRYL4KM7WVATTHQ, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.567093405 (574h50m11.43817644s) [noise=true , address=77.90.31.14:4661 , piece_id=IEB2DM4DEUUOVO23TPMAZBSS3ZQ6ZDALQSHPDG3PFVI3LRDCTYVA, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.567479400 (574h50m11.438562445s) [noise=true , address=145.79.227.33:4871 , piece_id=PQGO6LGUQRVZPY4SW5ZKSCWWD4XO3VIVRTOH2SR3WS6NC2PU7GHA, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.567940388 (574h50m11.439023433s) [noise=true , address=45.91.116.10:4084 , piece_id=OPFAROMJ72YAZBI5U5JUCS5QILYJBZAIUAZ6JCHHJ5RVYL24RGBA, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.569018561 (574h50m11.440101606s) [noise=true , address=77.90.31.19:4844 , piece_id=QIZWLWBW7ISNMX3M63E7GY4VF2W7A5YL7MDHPUA4RUDJA5JNBZYA, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.569331076 (574h50m11.440414111s) [noise=true , address=77.90.31.11:4553 , piece_id=L7JEGOLKSCEP7WN636L52M4FIHHR45CFTPHZ364PRKH5R7VFK4AA, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.570610432 (574h50m11.441693477s) [noise=true , address=145.79.227.6:4257 , piece_id=7CJ6G2FWM4R7M3UX3B43D7Q2W7FNE5JXECK5OJBGMFIH7XFDLDGQ, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.571211647 (574h50m11.442294692s) [noise=true , address=77.90.31.17:4770 , piece_id=2UM5EHD7LQ5Q4JDQQU25SSSZO3M42PA2ONFJCTSBV5YNPZUAMRWQ, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.571987254 (574h50m11.443070299s) [noise=true , address=45.91.116.11:4119 , piece_id=73IG2RUDUWJOGHPNQ6YSPASLHLWS5WDHR3GJCDAGLRBIN2LFBN7Q, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:11.576033158 (574h50m11.447116203s) [noise=true , address=145.79.227.19:4077 , piece_id=W4C6AHJW273EZ5MLGYH2SD3IZYU2SX6WFTYK74YXKWRPD4JL3VCQ, seg_pos={0 0}, upload=503934]: Uploading piece... Dec 29 14:19:13.279813157 (574h50m13.150896192s) [upload=503934]: Next returned nil segment Dec 29 14:19:13.279829327 (574h50m13.150912362s) [upload=503934]: Waiting for error group managing segments... Dec 29 14:19:13.560183063 (574h50m13.431266109s) [noise=true , address=145.79.227.3:4150 , piece_id=LMIOVDMWV7Y5QITOLUF25DDVJB5WAO3VDFMDUHB3BQDV4NUOOTGA, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.567861885 (574h50m13.43894493s) [noise=true , address=145.79.227.19:4077 , piece_id=W4C6AHJW273EZ5MLGYH2SD3IZYU2SX6WFTYK74YXKWRPD4JL3VCQ, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.570739594 (574h50m13.441822629s) [noise=true , address=145.79.227.6:4257 , piece_id=7CJ6G2FWM4R7M3UX3B43D7Q2W7FNE5JXECK5OJBGMFIH7XFDLDGQ, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.586389204 (574h50m13.457472239s) [noise=true , address=145.79.227.16:4909 , piece_id=3GRC7HPRMYQCR55UPCKZVXAQACRVFNY43YTT5HW4TRFSGDWVPECQ, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.739004535 (574h50m13.610087581s) [noise=true , address=77.90.31.19:4844 , piece_id=QIZWLWBW7ISNMX3M63E7GY4VF2W7A5YL7MDHPUA4RUDJA5JNBZYA, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.843073117 (574h50m13.714156163s) [noise=true , address=45.91.116.11:4119 , piece_id=73IG2RUDUWJOGHPNQ6YSPASLHLWS5WDHR3GJCDAGLRBIN2LFBN7Q, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.871648990 (574h50m13.742732025s) [noise=true , address=77.90.31.11:4553 , piece_id=L7JEGOLKSCEP7WN636L52M4FIHHR45CFTPHZ364PRKH5R7VFK4AA, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.872121721 (574h50m13.743204756s) [noise=true , address=77.90.31.14:4661 , piece_id=IEB2DM4DEUUOVO23TPMAZBSS3ZQ6ZDALQSHPDG3PFVI3LRDCTYVA, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.886765455 (574h50m13.7578485s) [noise=true , address=77.90.31.17:4770 , piece_id=2UM5EHD7LQ5Q4JDQQU25SSSZO3M42PA2ONFJCTSBV5YNPZUAMRWQ, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.936469859 (574h50m13.807552904s) [noise=true , address=45.91.116.8:4048 , piece_id=LJQBVZEUV2ZCZ4GIFADV2OZAMPSZMSWIS4AKQ5DJWLYAX5B4SXQQ, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.951237860 (574h50m13.822320905s) [noise=true , address=45.91.116.9:4261 , piece_id=MQ27XRRACEVZVUOC7MZLCFF4HBTB23BPMG7QPDH37J3VX7PXVQKQ, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:13.972366501 (574h50m13.843449566s) [noise=true , address=45.91.116.12:4227 , piece_id=CM3L7NS5DYZM3XAACV7JKNNPHYWWGXECBLKUPTRYL4KM7WVATTHQ, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:14.219075022 (574h50m14.090158067s) [noise=true , address=45.91.116.14:4189 , piece_id=Q7AV3TMIG4FOYUCFY3HOER5NOFGDNEN34PFD3PZ6EUIUQ2MZHMPA, seg_pos={0 0}, upload=503934]: Done uploading piece. err: <nil> Dec 29 14:19:14.219098155 (574h50m14.0901812s) [seg_pos={0 0}, upload=503934]: Segment reached optimal threshold of 13 pieces. Dec 29 14:19:14.219277987 (574h50m14.090361022s) [noise=true , address=77.90.31.13:4627 , piece_id=V6ASI4XTUW2ECBNZVCG373SEWDXVQN2EWNHHTK5RDMEGLLNIMD2A, seg_pos={0 0}, upload=503934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 29 14:19:14.219285352 (574h50m14.090368387s) [noise=true , address=77.90.31.13:4627 , piece_id=V6ASI4XTUW2ECBNZVCG373SEWDXVQN2EWNHHTK5RDMEGLLNIMD2A, seg_pos={0 0}, upload=503934]: Long tail cancellation due to optimal threshold Dec 29 14:19:14.219341899 (574h50m14.090424934s) [noise=true , address=145.79.227.33:4871 , piece_id=PQGO6LGUQRVZPY4SW5ZKSCWWD4XO3VIVRTOH2SR3WS6NC2PU7GHA, seg_pos={0 0}, upload=503934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 14:19:14.219354404 (574h50m14.090437439s) [noise=true , address=145.79.227.33:4871 , piece_id=PQGO6LGUQRVZPY4SW5ZKSCWWD4XO3VIVRTOH2SR3WS6NC2PU7GHA, seg_pos={0 0}, upload=503934]: Long tail cancellation due to optimal threshold Dec 29 14:19:14.219465816 (574h50m14.090548851s) [noise=true , address=45.91.116.10:4084 , piece_id=OPFAROMJ72YAZBI5U5JUCS5QILYJBZAIUAZ6JCHHJ5RVYL24RGBA, seg_pos={0 0}, upload=503934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 29 14:19:14.219471977 (574h50m14.090555022s) [noise=true , address=45.91.116.10:4084 , piece_id=OPFAROMJ72YAZBI5U5JUCS5QILYJBZAIUAZ6JCHHJ5RVYL24RGBA, seg_pos={0 0}, upload=503934]: Long tail cancellation due to optimal threshold Dec 29 14:19:14.219487917 (574h50m14.090570962s) [seg_pos={0 0}, upload=503934]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:19:14.219563221 (574h50m14.090646256s) [upload=503934]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:19:14.219568110 (574h50m14.090651146s) [upload=503934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:19:14.525973070 (574h50m14.397056105s) [upload=503934]: Done uploading segments... Dec 29 14:19:14.525992989 (574h50m14.397076034s) [upload=503934]: Upload finished. err: <nil> Duration: 30.002444064s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [......X...........X...X....X....X.XX...........X............] duration: 16.87577263s trace: https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd FAIL (2025-12-29 14:19:38 UTC) orbiter warning: context deadline exceeded Duration: 16.875794581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................XXX..X..XX...................] duration: 51.40610153s trace: https://jaeger.dev.storj.io/trace/266137264da4c90 FAIL (2025-12-29 14:19:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded ======LOGS====== NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments... Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces. Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments... Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil> Duration: 51.406136969s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.........X...X.X.XX....X........X.X............X..X.XX.XXX.X] duration: 30.001611971s trace: https://jaeger.dev.storj.io/trace/6afc00cfbd932405 FAIL (2025-12-29 14:19:09 UTC) orbiter warning: context deadline exceeded Duration: 30.001635477s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [.X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.XXXXX..] duration: 6.039629954s trace: https://jaeger.dev.storj.io/trace/20fb40b8f16de073 FAIL (2025-12-29 14:18:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7RFLNH34IRKYCJ2DY2X3AFYRWJBMGMOBNRIYMN5QFSDWQYX7UC2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: RKEYIEECGWJT6SZPSJN7TC7SJQGQJTAN2LA3RXH4IFQUFTD35NHQ) context deadline exceeded ======LOGS====== Dec 29 14:16:24.691919502 (574h47m24.563002537s) [upload=503893]: Starting upload Dec 29 14:16:24.691982702 (574h47m24.563065747s) [upload=503893]: Done starting upload Dec 29 14:16:24.692153067 (574h47m24.563236112s) [upload=503893]: Uploading segments... Dec 29 14:16:24.692194155 (574h47m24.5632772s) [upload=503893]: Got next segment. Inline: false Dec 29 14:16:24.692242267 (574h47m24.563325302s) [upload=503893]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:16:24.692249921 (574h47m24.563332967s) [upload=503893]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:16:24.938226046 (574h47m24.809309081s) [seg_pos={0 0}, upload=503893]: Begin upload segment... Dec 29 14:16:24.938284899 (574h47m24.809367944s) [seg_pos={0 0}, upload=503893]: Done begin upload segment. Dec 29 14:16:24.938337979 (574h47m24.809421024s) [noise=true , address=77.90.31.10:4520 , piece_id=IN5OWRRKE3CITFRJ72NTQOWM4FKRBR3SG65WCLCW6ZZ6M7U2MOZQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.938389097 (574h47m24.809472132s) [noise=true , address=45.91.116.11:4120 , piece_id=3K7NMLKSMFSR56QGWN5F67Q2A32ROO4CW75JVHV7BKOMRYQBW4ZQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.938956487 (574h47m24.810039522s) [noise=true , address=145.79.227.3:4150 , piece_id=CTIZX6QSUD4XJUFB764NTSNBW4VUXLPR4PBCGEXP7JRU7J2G6UHA, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.939300202 (574h47m24.810383237s) [noise=true , address=145.79.227.36:4517 , piece_id=KWR565P4N6G26KJFFWPI37WVR2UUQQOSBJKKSSL3FYJVMBRZNPCQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.939669075 (574h47m24.81075211s) [noise=true , address=45.91.116.5:4299 , piece_id=7RFLNH34IRKYCJ2DY2X3AFYRWJBMGMOBNRIYMN5QFSDWQYX7UC2A, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.940018159 (574h47m24.811101194s) [noise=true , address=45.91.116.4:4940 , piece_id=WPIGQE532RXQZIBVDPBLHHZZF7AQHWWTFPGPJ3NRKAUZ6RVLJCUA, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.940407150 (574h47m24.811490195s) [noise=true , address=145.79.227.10:4401 , piece_id=IBWBN4ENKA2SOQZWVZQLYW4V24ORSZKE4BEU5NILWOCDIFD44EDQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.940823623 (574h47m24.811906658s) [noise=true , address=145.79.227.7:4295 , piece_id=63LOTUXKTME7HNHD7HTAHA5L5KVQ6MZD4RMAPSAIZJCJGN2WQY6A, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.942214280 (574h47m24.813297315s) [noise=true , address=145.79.227.14:4840 , piece_id=BKB2I57O2MED47QJHWHSGYT3JPLLE4MQTBM4XZMA2SR4JBA3IVJA, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.942326364 (574h47m24.813409399s) [noise=true , address=145.79.227.21:4475 , piece_id=V2TFUPOKSWCWF3HMKSRGWHOJVNFQFXLIHGJD4RVY74GLJ6HXLOYQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.943567948 (574h47m24.814650993s) [noise=true , address=145.79.227.32:4834 , piece_id=IEUZYQAUGEOTKBKM2WUOFFHLAQBAF43EA3OURPYTD7B4OAXAINRQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.946151208 (574h47m24.817234243s) [noise=true , address=45.91.116.13:4154 , piece_id=RKEYIEECGWJT6SZPSJN7TC7SJQGQJTAN2LA3RXH4IFQUFTD35NHQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.948625589 (574h47m24.819708624s) [noise=true , address=45.91.116.8:4047 , piece_id=Y77WFPT23NBZE7EQH2P2DT56SLUF6B3WNM5GXIVM2FL5VXDXVVQQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.951686157 (574h47m24.822769192s) [noise=true , address=145.79.227.49:4339 , piece_id=LPN3NJA4YUONE54476J6XICUQEBJ23ESUD3FUYTQYA6F42CCEF3Q, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.953752742 (574h47m24.824835777s) [noise=true , address=145.79.227.46:4601 , piece_id=YNP7Y5OVAQ3LYWGIZLLTPWGCJ5PGJYK3CTOGPZXGNRTOGTN2QITQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.955071152 (574h47m24.826154198s) [noise=true , address=145.79.227.4:4187 , piece_id=KO5G3VO3QVSM42BMSTTZMZU35DKPKHCQKRHJSULALYTIXTOCCMLQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:26.637588345 (574h47m26.508671391s) [upload=503893]: Next returned nil segment Dec 29 14:16:26.637605217 (574h47m26.508688262s) [upload=503893]: Waiting for error group managing segments... Dec 29 14:16:26.917522541 (574h47m26.788605586s) [noise=true , address=145.79.227.10:4401 , piece_id=IBWBN4ENKA2SOQZWVZQLYW4V24ORSZKE4BEU5NILWOCDIFD44EDQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.925683291 (574h47m26.796766336s) [noise=true , address=145.79.227.21:4475 , piece_id=V2TFUPOKSWCWF3HMKSRGWHOJVNFQFXLIHGJD4RVY74GLJ6HXLOYQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.929644243 (574h47m26.800727288s) [noise=true , address=145.79.227.36:4517 , piece_id=KWR565P4N6G26KJFFWPI37WVR2UUQQOSBJKKSSL3FYJVMBRZNPCQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.938095024 (574h47m26.80917806s) [noise=true , address=145.79.227.32:4834 , piece_id=IEUZYQAUGEOTKBKM2WUOFFHLAQBAF43EA3OURPYTD7B4OAXAINRQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.941896353 (574h47m26.812979398s) [noise=true , address=145.79.227.3:4150 , piece_id=CTIZX6QSUD4XJUFB764NTSNBW4VUXLPR4PBCGEXP7JRU7J2G6UHA, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.954081435 (574h47m26.82516447s) [noise=true , address=145.79.227.7:4295 , piece_id=63LOTUXKTME7HNHD7HTAHA5L5KVQ6MZD4RMAPSAIZJCJGN2WQY6A, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.957620513 (574h47m26.828703549s) [noise=true , address=145.79.227.4:4187 , piece_id=KO5G3VO3QVSM42BMSTTZMZU35DKPKHCQKRHJSULALYTIXTOCCMLQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.962030882 (574h47m26.833113927s) [noise=true , address=145.79.227.46:4601 , piece_id=YNP7Y5OVAQ3LYWGIZLLTPWGCJ5PGJYK3CTOGPZXGNRTOGTN2QITQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.964667964 (574h47m26.835750999s) [noise=true , address=145.79.227.14:4840 , piece_id=BKB2I57O2MED47QJHWHSGYT3JPLLE4MQTBM4XZMA2SR4JBA3IVJA, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.199161473 (574h47m27.070244509s) [noise=true , address=45.91.116.13:4154 , piece_id=RKEYIEECGWJT6SZPSJN7TC7SJQGQJTAN2LA3RXH4IFQUFTD35NHQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.294506905 (574h47m27.16558994s) [noise=true , address=45.91.116.8:4047 , piece_id=Y77WFPT23NBZE7EQH2P2DT56SLUF6B3WNM5GXIVM2FL5VXDXVVQQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.309490366 (574h47m27.180573401s) [noise=true , address=45.91.116.4:4940 , piece_id=WPIGQE532RXQZIBVDPBLHHZZF7AQHWWTFPGPJ3NRKAUZ6RVLJCUA, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.912447706 (574h47m27.783530741s) [noise=true , address=45.91.116.5:4299 , piece_id=7RFLNH34IRKYCJ2DY2X3AFYRWJBMGMOBNRIYMN5QFSDWQYX7UC2A, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.912491359 (574h47m27.783574394s) [seg_pos={0 0}, upload=503893]: Segment reached optimal threshold of 13 pieces. Dec 29 14:16:27.912870872 (574h47m27.783953927s) [noise=true , address=145.79.227.49:4339 , piece_id=LPN3NJA4YUONE54476J6XICUQEBJ23ESUD3FUYTQYA6F42CCEF3Q, seg_pos={0 0}, upload=503893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 14:16:27.912882153 (574h47m27.783965198s) [noise=true , address=145.79.227.49:4339 , piece_id=LPN3NJA4YUONE54476J6XICUQEBJ23ESUD3FUYTQYA6F42CCEF3Q, seg_pos={0 0}, upload=503893]: Long tail cancellation due to optimal threshold Dec 29 14:16:27.913084018 (574h47m27.784167053s) [noise=true , address=45.91.116.11:4120 , piece_id=3K7NMLKSMFSR56QGWN5F67Q2A32ROO4CW75JVHV7BKOMRYQBW4ZQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 14:16:27.913095489 (574h47m27.784178524s) [noise=true , address=45.91.116.11:4120 , piece_id=3K7NMLKSMFSR56QGWN5F67Q2A32ROO4CW75JVHV7BKOMRYQBW4ZQ, seg_pos={0 0}, upload=503893]: Long tail cancellation due to optimal threshold Dec 29 14:16:27.913593026 (574h47m27.784676061s) [noise=true , address=77.90.31.10:4520 , piece_id=IN5OWRRKE3CITFRJ72NTQOWM4FKRBR3SG65WCLCW6ZZ6M7U2MOZQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 14:16:27.913603397 (574h47m27.784686442s) [noise=true , address=77.90.31.10:4520 , piece_id=IN5OWRRKE3CITFRJ72NTQOWM4FKRBR3SG65WCLCW6ZZ6M7U2MOZQ, seg_pos={0 0}, upload=503893]: Long tail cancellation due to optimal threshold Dec 29 14:16:27.913624136 (574h47m27.784707171s) [seg_pos={0 0}, upload=503893]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:16:27.913725859 (574h47m27.784808904s) [upload=503893]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:16:27.913734526 (574h47m27.784817561s) [upload=503893]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:16:28.216713763 (574h47m28.087796808s) [upload=503893]: Done uploading segments... Dec 29 14:16:28.216738911 (574h47m28.087821946s) [upload=503893]: Upload finished. err: <nil> Duration: 6.039660101s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................XXX..X..XX..................] duration: 44.218987872s trace: https://jaeger.dev.storj.io/trace/266137264da4c90 FAIL (2025-12-29 14:18:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded ======LOGS====== NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments... Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces. Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments... Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil> Duration: 44.219035272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [X.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001955446s trace: https://jaeger.dev.storj.io/trace/15290c8fcae3506 FAIL (2025-12-29 14:18:40 UTC) orbiter warning: context deadline exceeded Duration: 5.001972218s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.......X...........X...X....X....X.XX...........X...........] duration: 14.068200052s trace: https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd FAIL (2025-12-29 14:18:32 UTC) orbiter warning: context deadline exceeded Duration: 14.068229407s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..........X...X.X.XX....X........X.X............X..X.XX.XXX.] duration: 17.172924261s trace: https://jaeger.dev.storj.io/trace/311de7fb4d624684 FAIL (2025-12-29 14:18:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 17.172948546s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001360958s trace: https://jaeger.dev.storj.io/trace/38ff2ebc9677dc6e FAIL (2025-12-29 14:18:25 UTC) orbiter warning: context deadline exceeded Duration: 5.001376336s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration: 5.000661505s trace: https://jaeger.dev.storj.io/trace/24c943e2b39c9c95 FAIL (2025-12-29 14:17:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.0006847s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [........X...........X...X....X....X.XX...........X..........] duration: 16.492156651s trace: https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd FAIL (2025-12-29 14:17:47 UTC) orbiter warning: context deadline exceeded Duration: 16.492181117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [..X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.XXXXX.] duration: 5.978716133s trace: https://jaeger.dev.storj.io/trace/20fb40b8f16de073 FAIL (2025-12-29 14:17:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7RFLNH34IRKYCJ2DY2X3AFYRWJBMGMOBNRIYMN5QFSDWQYX7UC2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: RKEYIEECGWJT6SZPSJN7TC7SJQGQJTAN2LA3RXH4IFQUFTD35NHQ) context deadline exceeded ======LOGS====== Dec 29 14:16:24.691919502 (574h47m24.563002537s) [upload=503893]: Starting upload Dec 29 14:16:24.691982702 (574h47m24.563065747s) [upload=503893]: Done starting upload Dec 29 14:16:24.692153067 (574h47m24.563236112s) [upload=503893]: Uploading segments... Dec 29 14:16:24.692194155 (574h47m24.5632772s) [upload=503893]: Got next segment. Inline: false Dec 29 14:16:24.692242267 (574h47m24.563325302s) [upload=503893]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:16:24.692249921 (574h47m24.563332967s) [upload=503893]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:16:24.938226046 (574h47m24.809309081s) [seg_pos={0 0}, upload=503893]: Begin upload segment... Dec 29 14:16:24.938284899 (574h47m24.809367944s) [seg_pos={0 0}, upload=503893]: Done begin upload segment. Dec 29 14:16:24.938337979 (574h47m24.809421024s) [noise=true , address=77.90.31.10:4520 , piece_id=IN5OWRRKE3CITFRJ72NTQOWM4FKRBR3SG65WCLCW6ZZ6M7U2MOZQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.938389097 (574h47m24.809472132s) [noise=true , address=45.91.116.11:4120 , piece_id=3K7NMLKSMFSR56QGWN5F67Q2A32ROO4CW75JVHV7BKOMRYQBW4ZQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.938956487 (574h47m24.810039522s) [noise=true , address=145.79.227.3:4150 , piece_id=CTIZX6QSUD4XJUFB764NTSNBW4VUXLPR4PBCGEXP7JRU7J2G6UHA, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.939300202 (574h47m24.810383237s) [noise=true , address=145.79.227.36:4517 , piece_id=KWR565P4N6G26KJFFWPI37WVR2UUQQOSBJKKSSL3FYJVMBRZNPCQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.939669075 (574h47m24.81075211s) [noise=true , address=45.91.116.5:4299 , piece_id=7RFLNH34IRKYCJ2DY2X3AFYRWJBMGMOBNRIYMN5QFSDWQYX7UC2A, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.940018159 (574h47m24.811101194s) [noise=true , address=45.91.116.4:4940 , piece_id=WPIGQE532RXQZIBVDPBLHHZZF7AQHWWTFPGPJ3NRKAUZ6RVLJCUA, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.940407150 (574h47m24.811490195s) [noise=true , address=145.79.227.10:4401 , piece_id=IBWBN4ENKA2SOQZWVZQLYW4V24ORSZKE4BEU5NILWOCDIFD44EDQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.940823623 (574h47m24.811906658s) [noise=true , address=145.79.227.7:4295 , piece_id=63LOTUXKTME7HNHD7HTAHA5L5KVQ6MZD4RMAPSAIZJCJGN2WQY6A, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.942214280 (574h47m24.813297315s) [noise=true , address=145.79.227.14:4840 , piece_id=BKB2I57O2MED47QJHWHSGYT3JPLLE4MQTBM4XZMA2SR4JBA3IVJA, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.942326364 (574h47m24.813409399s) [noise=true , address=145.79.227.21:4475 , piece_id=V2TFUPOKSWCWF3HMKSRGWHOJVNFQFXLIHGJD4RVY74GLJ6HXLOYQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.943567948 (574h47m24.814650993s) [noise=true , address=145.79.227.32:4834 , piece_id=IEUZYQAUGEOTKBKM2WUOFFHLAQBAF43EA3OURPYTD7B4OAXAINRQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.946151208 (574h47m24.817234243s) [noise=true , address=45.91.116.13:4154 , piece_id=RKEYIEECGWJT6SZPSJN7TC7SJQGQJTAN2LA3RXH4IFQUFTD35NHQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.948625589 (574h47m24.819708624s) [noise=true , address=45.91.116.8:4047 , piece_id=Y77WFPT23NBZE7EQH2P2DT56SLUF6B3WNM5GXIVM2FL5VXDXVVQQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.951686157 (574h47m24.822769192s) [noise=true , address=145.79.227.49:4339 , piece_id=LPN3NJA4YUONE54476J6XICUQEBJ23ESUD3FUYTQYA6F42CCEF3Q, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.953752742 (574h47m24.824835777s) [noise=true , address=145.79.227.46:4601 , piece_id=YNP7Y5OVAQ3LYWGIZLLTPWGCJ5PGJYK3CTOGPZXGNRTOGTN2QITQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.955071152 (574h47m24.826154198s) [noise=true , address=145.79.227.4:4187 , piece_id=KO5G3VO3QVSM42BMSTTZMZU35DKPKHCQKRHJSULALYTIXTOCCMLQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:26.637588345 (574h47m26.508671391s) [upload=503893]: Next returned nil segment Dec 29 14:16:26.637605217 (574h47m26.508688262s) [upload=503893]: Waiting for error group managing segments... Dec 29 14:16:26.917522541 (574h47m26.788605586s) [noise=true , address=145.79.227.10:4401 , piece_id=IBWBN4ENKA2SOQZWVZQLYW4V24ORSZKE4BEU5NILWOCDIFD44EDQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.925683291 (574h47m26.796766336s) [noise=true , address=145.79.227.21:4475 , piece_id=V2TFUPOKSWCWF3HMKSRGWHOJVNFQFXLIHGJD4RVY74GLJ6HXLOYQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.929644243 (574h47m26.800727288s) [noise=true , address=145.79.227.36:4517 , piece_id=KWR565P4N6G26KJFFWPI37WVR2UUQQOSBJKKSSL3FYJVMBRZNPCQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.938095024 (574h47m26.80917806s) [noise=true , address=145.79.227.32:4834 , piece_id=IEUZYQAUGEOTKBKM2WUOFFHLAQBAF43EA3OURPYTD7B4OAXAINRQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.941896353 (574h47m26.812979398s) [noise=true , address=145.79.227.3:4150 , piece_id=CTIZX6QSUD4XJUFB764NTSNBW4VUXLPR4PBCGEXP7JRU7J2G6UHA, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.954081435 (574h47m26.82516447s) [noise=true , address=145.79.227.7:4295 , piece_id=63LOTUXKTME7HNHD7HTAHA5L5KVQ6MZD4RMAPSAIZJCJGN2WQY6A, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.957620513 (574h47m26.828703549s) [noise=true , address=145.79.227.4:4187 , piece_id=KO5G3VO3QVSM42BMSTTZMZU35DKPKHCQKRHJSULALYTIXTOCCMLQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.962030882 (574h47m26.833113927s) [noise=true , address=145.79.227.46:4601 , piece_id=YNP7Y5OVAQ3LYWGIZLLTPWGCJ5PGJYK3CTOGPZXGNRTOGTN2QITQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.964667964 (574h47m26.835750999s) [noise=true , address=145.79.227.14:4840 , piece_id=BKB2I57O2MED47QJHWHSGYT3JPLLE4MQTBM4XZMA2SR4JBA3IVJA, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.199161473 (574h47m27.070244509s) [noise=true , address=45.91.116.13:4154 , piece_id=RKEYIEECGWJT6SZPSJN7TC7SJQGQJTAN2LA3RXH4IFQUFTD35NHQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.294506905 (574h47m27.16558994s) [noise=true , address=45.91.116.8:4047 , piece_id=Y77WFPT23NBZE7EQH2P2DT56SLUF6B3WNM5GXIVM2FL5VXDXVVQQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.309490366 (574h47m27.180573401s) [noise=true , address=45.91.116.4:4940 , piece_id=WPIGQE532RXQZIBVDPBLHHZZF7AQHWWTFPGPJ3NRKAUZ6RVLJCUA, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.912447706 (574h47m27.783530741s) [noise=true , address=45.91.116.5:4299 , piece_id=7RFLNH34IRKYCJ2DY2X3AFYRWJBMGMOBNRIYMN5QFSDWQYX7UC2A, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.912491359 (574h47m27.783574394s) [seg_pos={0 0}, upload=503893]: Segment reached optimal threshold of 13 pieces. Dec 29 14:16:27.912870872 (574h47m27.783953927s) [noise=true , address=145.79.227.49:4339 , piece_id=LPN3NJA4YUONE54476J6XICUQEBJ23ESUD3FUYTQYA6F42CCEF3Q, seg_pos={0 0}, upload=503893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 14:16:27.912882153 (574h47m27.783965198s) [noise=true , address=145.79.227.49:4339 , piece_id=LPN3NJA4YUONE54476J6XICUQEBJ23ESUD3FUYTQYA6F42CCEF3Q, seg_pos={0 0}, upload=503893]: Long tail cancellation due to optimal threshold Dec 29 14:16:27.913084018 (574h47m27.784167053s) [noise=true , address=45.91.116.11:4120 , piece_id=3K7NMLKSMFSR56QGWN5F67Q2A32ROO4CW75JVHV7BKOMRYQBW4ZQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 14:16:27.913095489 (574h47m27.784178524s) [noise=true , address=45.91.116.11:4120 , piece_id=3K7NMLKSMFSR56QGWN5F67Q2A32ROO4CW75JVHV7BKOMRYQBW4ZQ, seg_pos={0 0}, upload=503893]: Long tail cancellation due to optimal threshold Dec 29 14:16:27.913593026 (574h47m27.784676061s) [noise=true , address=77.90.31.10:4520 , piece_id=IN5OWRRKE3CITFRJ72NTQOWM4FKRBR3SG65WCLCW6ZZ6M7U2MOZQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 14:16:27.913603397 (574h47m27.784686442s) [noise=true , address=77.90.31.10:4520 , piece_id=IN5OWRRKE3CITFRJ72NTQOWM4FKRBR3SG65WCLCW6ZZ6M7U2MOZQ, seg_pos={0 0}, upload=503893]: Long tail cancellation due to optimal threshold Dec 29 14:16:27.913624136 (574h47m27.784707171s) [seg_pos={0 0}, upload=503893]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:16:27.913725859 (574h47m27.784808904s) [upload=503893]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:16:27.913734526 (574h47m27.784817561s) [upload=503893]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:16:28.216713763 (574h47m28.087796808s) [upload=503893]: Done uploading segments... Dec 29 14:16:28.216738911 (574h47m28.087821946s) [upload=503893]: Upload finished. err: <nil> Duration: 5.978740721s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................XXX..X..XX.................] duration: 37.840608511s trace: https://jaeger.dev.storj.io/trace/266137264da4c90 FAIL (2025-12-29 14:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded ======LOGS====== NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments... Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces. Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments... Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil> Duration: 37.840639059s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.........X...........X...X....X....X.XX...........X.........] duration: 13.817505308s trace: https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd FAIL (2025-12-29 14:17:20 UTC) orbiter warning: context deadline exceeded Duration: 13.817531567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration: 5.001337922s trace: https://jaeger.dev.storj.io/trace/3f5b93e17b8b231b FAIL (2025-12-29 14:17:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001362248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...........X...X.X.XX....X........X.X............X..X.XX.XXX] duration: 10.819357579s trace: https://jaeger.dev.storj.io/trace/311de7fb4d624684 FAIL (2025-12-29 14:17:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.819382856s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 5.001667959s trace: https://jaeger.dev.storj.io/trace/7e1ce8acabee11df FAIL (2025-12-29 14:17:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001693819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.XXXXX] duration: 30.013486152s trace: https://jaeger.dev.storj.io/trace/20fb40b8f16de073 FAIL (2025-12-29 14:16:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 7RFLNH34IRKYCJ2DY2X3AFYRWJBMGMOBNRIYMN5QFSDWQYX7UC2A) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: RKEYIEECGWJT6SZPSJN7TC7SJQGQJTAN2LA3RXH4IFQUFTD35NHQ) context deadline exceeded ======LOGS====== Dec 29 14:16:24.691919502 (574h47m24.563002537s) [upload=503893]: Starting upload Dec 29 14:16:24.691982702 (574h47m24.563065747s) [upload=503893]: Done starting upload Dec 29 14:16:24.692153067 (574h47m24.563236112s) [upload=503893]: Uploading segments... Dec 29 14:16:24.692194155 (574h47m24.5632772s) [upload=503893]: Got next segment. Inline: false Dec 29 14:16:24.692242267 (574h47m24.563325302s) [upload=503893]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:16:24.692249921 (574h47m24.563332967s) [upload=503893]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:16:24.938226046 (574h47m24.809309081s) [seg_pos={0 0}, upload=503893]: Begin upload segment... Dec 29 14:16:24.938284899 (574h47m24.809367944s) [seg_pos={0 0}, upload=503893]: Done begin upload segment. Dec 29 14:16:24.938337979 (574h47m24.809421024s) [noise=true , address=77.90.31.10:4520 , piece_id=IN5OWRRKE3CITFRJ72NTQOWM4FKRBR3SG65WCLCW6ZZ6M7U2MOZQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.938389097 (574h47m24.809472132s) [noise=true , address=45.91.116.11:4120 , piece_id=3K7NMLKSMFSR56QGWN5F67Q2A32ROO4CW75JVHV7BKOMRYQBW4ZQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.938956487 (574h47m24.810039522s) [noise=true , address=145.79.227.3:4150 , piece_id=CTIZX6QSUD4XJUFB764NTSNBW4VUXLPR4PBCGEXP7JRU7J2G6UHA, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.939300202 (574h47m24.810383237s) [noise=true , address=145.79.227.36:4517 , piece_id=KWR565P4N6G26KJFFWPI37WVR2UUQQOSBJKKSSL3FYJVMBRZNPCQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.939669075 (574h47m24.81075211s) [noise=true , address=45.91.116.5:4299 , piece_id=7RFLNH34IRKYCJ2DY2X3AFYRWJBMGMOBNRIYMN5QFSDWQYX7UC2A, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.940018159 (574h47m24.811101194s) [noise=true , address=45.91.116.4:4940 , piece_id=WPIGQE532RXQZIBVDPBLHHZZF7AQHWWTFPGPJ3NRKAUZ6RVLJCUA, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.940407150 (574h47m24.811490195s) [noise=true , address=145.79.227.10:4401 , piece_id=IBWBN4ENKA2SOQZWVZQLYW4V24ORSZKE4BEU5NILWOCDIFD44EDQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.940823623 (574h47m24.811906658s) [noise=true , address=145.79.227.7:4295 , piece_id=63LOTUXKTME7HNHD7HTAHA5L5KVQ6MZD4RMAPSAIZJCJGN2WQY6A, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.942214280 (574h47m24.813297315s) [noise=true , address=145.79.227.14:4840 , piece_id=BKB2I57O2MED47QJHWHSGYT3JPLLE4MQTBM4XZMA2SR4JBA3IVJA, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.942326364 (574h47m24.813409399s) [noise=true , address=145.79.227.21:4475 , piece_id=V2TFUPOKSWCWF3HMKSRGWHOJVNFQFXLIHGJD4RVY74GLJ6HXLOYQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.943567948 (574h47m24.814650993s) [noise=true , address=145.79.227.32:4834 , piece_id=IEUZYQAUGEOTKBKM2WUOFFHLAQBAF43EA3OURPYTD7B4OAXAINRQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.946151208 (574h47m24.817234243s) [noise=true , address=45.91.116.13:4154 , piece_id=RKEYIEECGWJT6SZPSJN7TC7SJQGQJTAN2LA3RXH4IFQUFTD35NHQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.948625589 (574h47m24.819708624s) [noise=true , address=45.91.116.8:4047 , piece_id=Y77WFPT23NBZE7EQH2P2DT56SLUF6B3WNM5GXIVM2FL5VXDXVVQQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.951686157 (574h47m24.822769192s) [noise=true , address=145.79.227.49:4339 , piece_id=LPN3NJA4YUONE54476J6XICUQEBJ23ESUD3FUYTQYA6F42CCEF3Q, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.953752742 (574h47m24.824835777s) [noise=true , address=145.79.227.46:4601 , piece_id=YNP7Y5OVAQ3LYWGIZLLTPWGCJ5PGJYK3CTOGPZXGNRTOGTN2QITQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:24.955071152 (574h47m24.826154198s) [noise=true , address=145.79.227.4:4187 , piece_id=KO5G3VO3QVSM42BMSTTZMZU35DKPKHCQKRHJSULALYTIXTOCCMLQ, seg_pos={0 0}, upload=503893]: Uploading piece... Dec 29 14:16:26.637588345 (574h47m26.508671391s) [upload=503893]: Next returned nil segment Dec 29 14:16:26.637605217 (574h47m26.508688262s) [upload=503893]: Waiting for error group managing segments... Dec 29 14:16:26.917522541 (574h47m26.788605586s) [noise=true , address=145.79.227.10:4401 , piece_id=IBWBN4ENKA2SOQZWVZQLYW4V24ORSZKE4BEU5NILWOCDIFD44EDQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.925683291 (574h47m26.796766336s) [noise=true , address=145.79.227.21:4475 , piece_id=V2TFUPOKSWCWF3HMKSRGWHOJVNFQFXLIHGJD4RVY74GLJ6HXLOYQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.929644243 (574h47m26.800727288s) [noise=true , address=145.79.227.36:4517 , piece_id=KWR565P4N6G26KJFFWPI37WVR2UUQQOSBJKKSSL3FYJVMBRZNPCQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.938095024 (574h47m26.80917806s) [noise=true , address=145.79.227.32:4834 , piece_id=IEUZYQAUGEOTKBKM2WUOFFHLAQBAF43EA3OURPYTD7B4OAXAINRQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.941896353 (574h47m26.812979398s) [noise=true , address=145.79.227.3:4150 , piece_id=CTIZX6QSUD4XJUFB764NTSNBW4VUXLPR4PBCGEXP7JRU7J2G6UHA, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.954081435 (574h47m26.82516447s) [noise=true , address=145.79.227.7:4295 , piece_id=63LOTUXKTME7HNHD7HTAHA5L5KVQ6MZD4RMAPSAIZJCJGN2WQY6A, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.957620513 (574h47m26.828703549s) [noise=true , address=145.79.227.4:4187 , piece_id=KO5G3VO3QVSM42BMSTTZMZU35DKPKHCQKRHJSULALYTIXTOCCMLQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.962030882 (574h47m26.833113927s) [noise=true , address=145.79.227.46:4601 , piece_id=YNP7Y5OVAQ3LYWGIZLLTPWGCJ5PGJYK3CTOGPZXGNRTOGTN2QITQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:26.964667964 (574h47m26.835750999s) [noise=true , address=145.79.227.14:4840 , piece_id=BKB2I57O2MED47QJHWHSGYT3JPLLE4MQTBM4XZMA2SR4JBA3IVJA, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.199161473 (574h47m27.070244509s) [noise=true , address=45.91.116.13:4154 , piece_id=RKEYIEECGWJT6SZPSJN7TC7SJQGQJTAN2LA3RXH4IFQUFTD35NHQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.294506905 (574h47m27.16558994s) [noise=true , address=45.91.116.8:4047 , piece_id=Y77WFPT23NBZE7EQH2P2DT56SLUF6B3WNM5GXIVM2FL5VXDXVVQQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.309490366 (574h47m27.180573401s) [noise=true , address=45.91.116.4:4940 , piece_id=WPIGQE532RXQZIBVDPBLHHZZF7AQHWWTFPGPJ3NRKAUZ6RVLJCUA, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.912447706 (574h47m27.783530741s) [noise=true , address=45.91.116.5:4299 , piece_id=7RFLNH34IRKYCJ2DY2X3AFYRWJBMGMOBNRIYMN5QFSDWQYX7UC2A, seg_pos={0 0}, upload=503893]: Done uploading piece. err: <nil> Dec 29 14:16:27.912491359 (574h47m27.783574394s) [seg_pos={0 0}, upload=503893]: Segment reached optimal threshold of 13 pieces. Dec 29 14:16:27.912870872 (574h47m27.783953927s) [noise=true , address=145.79.227.49:4339 , piece_id=LPN3NJA4YUONE54476J6XICUQEBJ23ESUD3FUYTQYA6F42CCEF3Q, seg_pos={0 0}, upload=503893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 14:16:27.912882153 (574h47m27.783965198s) [noise=true , address=145.79.227.49:4339 , piece_id=LPN3NJA4YUONE54476J6XICUQEBJ23ESUD3FUYTQYA6F42CCEF3Q, seg_pos={0 0}, upload=503893]: Long tail cancellation due to optimal threshold Dec 29 14:16:27.913084018 (574h47m27.784167053s) [noise=true , address=45.91.116.11:4120 , piece_id=3K7NMLKSMFSR56QGWN5F67Q2A32ROO4CW75JVHV7BKOMRYQBW4ZQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 29 14:16:27.913095489 (574h47m27.784178524s) [noise=true , address=45.91.116.11:4120 , piece_id=3K7NMLKSMFSR56QGWN5F67Q2A32ROO4CW75JVHV7BKOMRYQBW4ZQ, seg_pos={0 0}, upload=503893]: Long tail cancellation due to optimal threshold Dec 29 14:16:27.913593026 (574h47m27.784676061s) [noise=true , address=77.90.31.10:4520 , piece_id=IN5OWRRKE3CITFRJ72NTQOWM4FKRBR3SG65WCLCW6ZZ6M7U2MOZQ, seg_pos={0 0}, upload=503893]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 29 14:16:27.913603397 (574h47m27.784686442s) [noise=true , address=77.90.31.10:4520 , piece_id=IN5OWRRKE3CITFRJ72NTQOWM4FKRBR3SG65WCLCW6ZZ6M7U2MOZQ, seg_pos={0 0}, upload=503893]: Long tail cancellation due to optimal threshold Dec 29 14:16:27.913624136 (574h47m27.784707171s) [seg_pos={0 0}, upload=503893]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:16:27.913725859 (574h47m27.784808904s) [upload=503893]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:16:27.913734526 (574h47m27.784817561s) [upload=503893]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:16:28.216713763 (574h47m28.087796808s) [upload=503893]: Done uploading segments... Dec 29 14:16:28.216738911 (574h47m28.087821946s) [upload=503893]: Upload finished. err: <nil> Duration: 30.013538984s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..........X...........X...X....X....X.XX...........X........] duration: 15.472285662s trace: https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd FAIL (2025-12-29 14:16:52 UTC) orbiter warning: context deadline exceeded Duration: 15.472313416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [............X...X.X.XX....X........X.X............X..X.XX.XX] duration: 30.001116557s trace: https://jaeger.dev.storj.io/trace/475f28dbb71867b8 FAIL (2025-12-29 14:16:38 UTC) orbiter warning: context deadline exceeded Duration: 30.001170148s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 5.001417448s trace: https://jaeger.dev.storj.io/trace/626674cd298f3698 FAIL (2025-12-29 14:16:36 UTC) orbiter warning: context deadline exceeded Duration: 5.001439661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................XXX..X..XX................] duration: 15.920362573s trace: https://jaeger.dev.storj.io/trace/266137264da4c90 FAIL (2025-12-29 14:16:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded ======LOGS====== NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments... Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces. Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments... Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil> Duration: 15.920379745s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........X...........X...X....X....X.XX...........X.......] duration: 24.38267038s trace: https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd FAIL (2025-12-29 14:16:10 UTC) orbiter warning: context deadline exceeded Duration: 24.382736856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................XXX..X..XX...............] duration: 52.052413479s trace: https://jaeger.dev.storj.io/trace/266137264da4c90 FAIL (2025-12-29 14:16:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded ======LOGS====== NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece... Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments... Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil> Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces. Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments... Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil> Duration: 52.052444086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.XXXX] duration: 30.004838979s trace: https://jaeger.dev.storj.io/trace/10193fc2eb32c947 FAIL (2025-12-29 14:16:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWOH5QTHLQLEKNLATC2HIWVOUJEMZKANPPXIS552WLICVYLN7LBA) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: NIKJXWBBU52GXSNBM6JDS2HRLVGFJTNM2JTGVYNGU6R23VX27AKA) context deadline exceeded; piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: JHD47RFW6BNBTDQYM4HLOGQRB7XZ3TOW6C33LKVVNSWIQRZNQNVA) context deadline exceeded ======LOGS====== ments... Dec 29 14:15:33.260271115 (574h46m33.13135415s) [upload=503882]: Got next segment. Inline: false Dec 29 14:15:33.260316872 (574h46m33.131399907s) [upload=503882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:15:33.260323294 (574h46m33.131406339s) [upload=503882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:15:33.506382327 (574h46m33.377465362s) [seg_pos={0 0}, upload=503882]: Begin upload segment... Dec 29 14:15:33.506454234 (574h46m33.377537279s) [seg_pos={0 0}, upload=503882]: Done begin upload segment. Dec 29 14:15:33.506493228 (574h46m33.377576263s) [noise=true , address=77.90.31.16:4735 , piece_id=LPCIHLW6MQOE5HOW7S5TSEDAZIP2IGUTDA6MYIX5XPHKESCGUTSA, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.506506985 (574h46m33.37759002s) [noise=true , address=145.79.227.49:4339 , piece_id=LB2MSGYDZCSIH5AKJ4CGMJGFICV4QSHPQM3B457BWEU3MWAVYPBQ, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.506566898 (574h46m33.377649944s) [noise=true , address=45.91.116.8:4045 , piece_id=EWOH5QTHLQLEKNLATC2HIWVOUJEMZKANPPXIS552WLICVYLN7LBA, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.506751390 (574h46m33.377834436s) [noise=true , address=145.79.227.50:4373 , piece_id=A2SIPTXTCALI26PBJTVMYR2RR4NY465QTBNUN5ITIBCLRIQZL2ZA, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.507287140 (574h46m33.378370185s) [noise=true , address=77.90.31.10:4520 , piece_id=MNGAWWQ2ADS7VRCDW6D6MPF5IKXUUDAALIJTYCFFGAJRJESMCA5A, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.508111330 (574h46m33.379194375s) [noise=true , address=77.90.31.13:4626 , piece_id=OKTR46G42ZM7IWGOEXEUNGZKGQLL3NSDVJQMYICOH6MKU3RNEBPA, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.508644946 (574h46m33.379727991s) [noise=true , address=77.90.31.14:4663 , piece_id=NIKJXWBBU52GXSNBM6JDS2HRLVGFJTNM2JTGVYNGU6R23VX27AKA, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.509306786 (574h46m33.380389831s) [noise=true , address=145.79.227.39:4013 , piece_id=3ESUM7YP4CXERXIDZTN7AYCKTJ2DIEQPSJTZOEGCML3USRF3OH7A, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.510466123 (574h46m33.381549168s) [noise=true , address=145.79.227.42:4121 , piece_id=P4SIVWDYWUASATDL22XXLQ7NPQGVKMITW5UYOUHFOLBZLL3DDURA, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.511091885 (574h46m33.38217492s) [noise=true , address=145.79.227.2:4113 , piece_id=2X6XG7C4EPOLXBRJAERLCQDXM6WPMMNNCJ35YOR7QG7KEAY5EAWQ, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.511724480 (574h46m33.382807515s) [noise=true , address=145.79.227.52:4445 , piece_id=SJPV4P27AMVECJTCP4EAAQ4RBWERKDSFLY5JXCOBY6LHXQPCL3BQ, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.512177302 (574h46m33.383260348s) [noise=true , address=45.91.116.6:4976 , piece_id=XU5CEZXYWFIRTRXCZ3UFWED52FTAJ2UYMJXARYABSIPTK6XTJV3A, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.512562336 (574h46m33.383645371s) [noise=true , address=45.91.116.4:4940 , piece_id=2UNC7ZKL5O7VLOGZ63G6IT6GY75KAMEGLCTYRCJUHH4DG4G6RFRQ, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.513070773 (574h46m33.384153808s) [noise=true , address=77.90.31.17:4770 , piece_id=JHD47RFW6BNBTDQYM4HLOGQRB7XZ3TOW6C33LKVVNSWIQRZNQNVA, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.513545437 (574h46m33.384628472s) [noise=true , address=145.79.227.6:4257 , piece_id=4HWWRJF72ZOWBNQ2IEEPQAOKSUXZJUHPMDRJIF65O3QILDKSVA2Q, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:33.513845859 (574h46m33.384928934s) [noise=true , address=45.91.116.10:4081 , piece_id=5R66S7D4R2KOHFIZGRVCDGUBHQRQGAMLNFA7WE44MU4FR7FOSYYA, seg_pos={0 0}, upload=503882]: Uploading piece... Dec 29 14:15:35.203785045 (574h46m35.07486808s) [upload=503882]: Next returned nil segment Dec 29 14:15:35.203798611 (574h46m35.074881646s) [upload=503882]: Waiting for error group managing segments... Dec 29 14:15:35.489523164 (574h46m35.360606199s) [noise=true , address=145.79.227.6:4257 , piece_id=4HWWRJF72ZOWBNQ2IEEPQAOKSUXZJUHPMDRJIF65O3QILDKSVA2Q, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:35.496864953 (574h46m35.367947998s) [noise=true , address=145.79.227.2:4113 , piece_id=2X6XG7C4EPOLXBRJAERLCQDXM6WPMMNNCJ35YOR7QG7KEAY5EAWQ, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:35.500882103 (574h46m35.371965148s) [noise=true , address=145.79.227.39:4013 , piece_id=3ESUM7YP4CXERXIDZTN7AYCKTJ2DIEQPSJTZOEGCML3USRF3OH7A, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:35.509006112 (574h46m35.380089147s) [noise=true , address=145.79.227.42:4121 , piece_id=P4SIVWDYWUASATDL22XXLQ7NPQGVKMITW5UYOUHFOLBZLL3DDURA, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:35.520031696 (574h46m35.391114731s) [noise=true , address=145.79.227.52:4445 , piece_id=SJPV4P27AMVECJTCP4EAAQ4RBWERKDSFLY5JXCOBY6LHXQPCL3BQ, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:35.532555914 (574h46m35.403638949s) [noise=true , address=145.79.227.50:4373 , piece_id=A2SIPTXTCALI26PBJTVMYR2RR4NY465QTBNUN5ITIBCLRIQZL2ZA, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:35.806147824 (574h46m35.677230869s) [noise=true , address=77.90.31.14:4663 , piece_id=NIKJXWBBU52GXSNBM6JDS2HRLVGFJTNM2JTGVYNGU6R23VX27AKA, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:35.815991738 (574h46m35.687074783s) [noise=true , address=45.91.116.8:4045 , piece_id=EWOH5QTHLQLEKNLATC2HIWVOUJEMZKANPPXIS552WLICVYLN7LBA, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:36.013165186 (574h46m35.884248231s) [noise=true , address=77.90.31.16:4735 , piece_id=LPCIHLW6MQOE5HOW7S5TSEDAZIP2IGUTDA6MYIX5XPHKESCGUTSA, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:36.032885917 (574h46m35.903968952s) [noise=true , address=77.90.31.17:4770 , piece_id=JHD47RFW6BNBTDQYM4HLOGQRB7XZ3TOW6C33LKVVNSWIQRZNQNVA, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:36.557080881 (574h46m36.428163927s) [noise=true , address=45.91.116.4:4940 , piece_id=2UNC7ZKL5O7VLOGZ63G6IT6GY75KAMEGLCTYRCJUHH4DG4G6RFRQ, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:37.105432936 (574h46m36.976515981s) [noise=true , address=145.79.227.49:4339 , piece_id=LB2MSGYDZCSIH5AKJ4CGMJGFICV4QSHPQM3B457BWEU3MWAVYPBQ, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:37.139448968 (574h46m37.010532013s) [noise=true , address=77.90.31.10:4520 , piece_id=MNGAWWQ2ADS7VRCDW6D6MPF5IKXUUDAALIJTYCFFGAJRJESMCA5A, seg_pos={0 0}, upload=503882]: Done uploading piece. err: <nil> Dec 29 14:15:37.139470188 (574h46m37.010553233s) [seg_pos={0 0}, upload=503882]: Segment reached optimal threshold of 13 pieces. Dec 29 14:15:37.139664288 (574h46m37.010747333s) [noise=true , address=45.91.116.10:4081 , piece_id=5R66S7D4R2KOHFIZGRVCDGUBHQRQGAMLNFA7WE44MU4FR7FOSYYA, seg_pos={0 0}, upload=503882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:51730->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:51730->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51730->45.91.116.10:4081: use of closed network connection Dec 29 14:15:37.139670149 (574h46m37.010753184s) [noise=true , address=45.91.116.10:4081 , piece_id=5R66S7D4R2KOHFIZGRVCDGUBHQRQGAMLNFA7WE44MU4FR7FOSYYA, seg_pos={0 0}, upload=503882]: Long tail cancellation due to optimal threshold Dec 29 14:15:37.139903804 (574h46m37.010986839s) [noise=true , address=45.91.116.6:4976 , piece_id=XU5CEZXYWFIRTRXCZ3UFWED52FTAJ2UYMJXARYABSIPTK6XTJV3A, seg_pos={0 0}, upload=503882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:47690->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:47690->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47690->45.91.116.6:4976: use of closed network connection Dec 29 14:15:37.139911338 (574h46m37.010994373s) [noise=true , address=45.91.116.6:4976 , piece_id=XU5CEZXYWFIRTRXCZ3UFWED52FTAJ2UYMJXARYABSIPTK6XTJV3A, seg_pos={0 0}, upload=503882]: Long tail cancellation due to optimal threshold Dec 29 14:15:37.139922189 (574h46m37.011005224s) [noise=true , address=77.90.31.13:4626 , piece_id=OKTR46G42ZM7IWGOEXEUNGZKGQLL3NSDVJQMYICOH6MKU3RNEBPA, seg_pos={0 0}, upload=503882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 14:15:37.139938410 (574h46m37.011021445s) [noise=true , address=77.90.31.13:4626 , piece_id=OKTR46G42ZM7IWGOEXEUNGZKGQLL3NSDVJQMYICOH6MKU3RNEBPA, seg_pos={0 0}, upload=503882]: Long tail cancellation due to optimal threshold Dec 29 14:15:37.139967875 (574h46m37.011050921s) [seg_pos={0 0}, upload=503882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:15:37.140094517 (574h46m37.011177552s) [upload=503882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:15:37.140102853 (574h46m37.011185898s) [upload=503882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:15:37.430701185 (574h46m37.30178423s) [upload=503882]: Done uploading segments... Dec 29 14:15:37.430720261 (574h46m37.301803306s) [upload=503882]: Upload finished. err: <nil> Duration: 30.004852967s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration: 5.000840332s trace: https://jaeger.dev.storj.io/trace/41730f3c362cfffb FAIL (2025-12-29 14:15:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000865861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............X...X.X.XX....X........X.X............X..X.XX.X] duration: 30.001196878s trace: https://jaeger.dev.storj.io/trace/6987bddee8955b2a FAIL (2025-12-29 14:15:47 UTC) orbiter warning: context deadline exceeded Duration: 30.001221866s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [............X...........X...X....X....X.XX...........X......] duration: 23.967981933s trace: https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd FAIL (2025-12-29 14:15:38 UTC) orbiter warning: context deadline exceeded Duration: 23.968009135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.XXX] duration: 30.003724133s trace: https://jaeger.dev.storj.io/trace/3b0862ff747579ec FAIL (2025-12-29 14:15:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: JZYNINGL7O6FRK2S324MDHAPRM2W3AUAAEHWLTBONFOHZ4CAGCXQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 3SHNXE7SWJKBHDTYFBLA2FYYBD3LLJ35DXRLQF2NX5EKDL4RTENA) context deadline exceeded ======LOGS====== Dec 29 14:15:03.042575357 (574h46m2.913658492s) [upload=503874]: Starting upload Dec 29 14:15:03.042658144 (574h46m2.913741199s) [upload=503874]: Done starting upload Dec 29 14:15:03.042928900 (574h46m2.914011945s) [upload=503874]: Uploading segments... Dec 29 14:15:03.043041144 (574h46m2.914124199s) [upload=503874]: Got next segment. Inline: false Dec 29 14:15:03.043139341 (574h46m2.914222396s) [upload=503874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 14:15:03.043161323 (574h46m2.914244368s) [upload=503874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 14:15:03.293033908 (574h46m3.164116983s) [seg_pos={0 0}, upload=503874]: Begin upload segment... Dec 29 14:15:03.293103680 (574h46m3.164186745s) [seg_pos={0 0}, upload=503874]: Done begin upload segment. Dec 29 14:15:03.293138897 (574h46m3.164221983s) [noise=true , address=145.79.227.10:4401 , piece_id=AMICSV3TCEZIC77BS7EUFSGOQ6XQODJT4G5UPAUXN2IF5QHC6PZQ, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.293179004 (574h46m3.164262069s) [noise=true , address=145.79.227.11:4729 , piece_id=4NZIKCEZ37GMZMQ2KUEQCFAQ3PBTH7S5R4LOGRSJUPGT7HVBUJWA, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.294153139 (574h46m3.165236194s) [noise=true , address=45.91.116.11:4120 , piece_id=C5G7SX3HIBNIOS4F4VQWSAO6MLITBKSIJH5BPKWLVB5X4DXQN3ZQ, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.294734476 (574h46m3.165817521s) [noise=true , address=145.79.227.17:4970 , piece_id=FR3NESBNJLG7F5DDCSYDNAASTKY2S3Z5GE4JODJ25X77DLBVZAKQ, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.295214190 (574h46m3.166297235s) [noise=true , address=45.91.116.13:4153 , piece_id=N7OCYCLSMTGXC6JGXQBPILPTPMAO6H4GNLQ5BHOAQRNXUGROIY5Q, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.295698422 (574h46m3.166781467s) [noise=true , address=145.79.227.39:4015 , piece_id=J4HS6HXM3ZYQGIL6JSYKFIDGUWRRX5C6HBSB2OWDDXILB7OJI77A, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.296213592 (574h46m3.167296637s) [noise=true , address=77.90.31.16:4735 , piece_id=JZYNINGL7O6FRK2S324MDHAPRM2W3AUAAEHWLTBONFOHZ4CAGCXQ, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.296709417 (574h46m3.167792462s) [noise=true , address=145.79.227.46:4603 , piece_id=YLBY54LNV3O3NRKWIBFCTHDZVSXDH2M7AGVHRKSGOF6EKIEMC3OQ, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.297206222 (574h46m3.168289268s) [noise=true , address=77.90.31.14:4663 , piece_id=3SHNXE7SWJKBHDTYFBLA2FYYBD3LLJ35DXRLQF2NX5EKDL4RTENA, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.297752343 (574h46m3.168835388s) [noise=true , address=145.79.227.30:4761 , piece_id=VLQUQQK64ELKAVLZOCW6DIQVG4RHVUDCHQ665RWJY3UDZCLKNWRA, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.298292982 (574h46m3.169376027s) [noise=true , address=145.79.227.6:4257 , piece_id=BOD75UC7DFIJ7H47EP72YN6LHRPMKLJLMXHFHKUPYH56BH7JCB6Q, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.298804957 (574h46m3.169888062s) [noise=true , address=145.79.227.18:4006 , piece_id=34XCQVUGN5SRUENEMF232ZFYYFZSJKWUP636PK5LVI5KWZZFYPKQ, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.298876443 (574h46m3.169959488s) [noise=true , address=45.91.116.6:4974 , piece_id=53DHBHYME6SXPQU2YRA4VAKDTPEFLZNV62P4G6KQ54QQUBPIDYJQ, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.299412463 (574h46m3.170495508s) [noise=true , address=145.79.227.44:4195 , piece_id=T2GAKUTG3RERSAD6CTOVH6AWTNIQD2UNI5JFXXIB4W3YFSUDREFA, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.299612584 (574h46m3.17069563s) [noise=true , address=45.91.116.7:4011 , piece_id=QSM5QVYA4NDTCXHWS42MIRYVYGYAUKS7F6DZELUKCWUOHPDU6CRA, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:03.299974213 (574h46m3.171057268s) [noise=true , address=145.79.227.13:4804 , piece_id=LC2R4BSYDRRTJEZGYKHV32C67HVS6VQ6CFT6CATUOJLMPX2F73AA, seg_pos={0 0}, upload=503874]: Uploading piece... Dec 29 14:15:04.947045912 (574h46m4.818128947s) [upload=503874]: Next returned nil segment Dec 29 14:15:04.947064237 (574h46m4.818147282s) [upload=503874]: Waiting for error group managing segments... Dec 29 14:15:05.319939467 (574h46m5.191022512s) [noise=true , address=145.79.227.10:4401 , piece_id=AMICSV3TCEZIC77BS7EUFSGOQ6XQODJT4G5UPAUXN2IF5QHC6PZQ, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.347654642 (574h46m5.218737687s) [noise=true , address=145.79.227.18:4006 , piece_id=34XCQVUGN5SRUENEMF232ZFYYFZSJKWUP636PK5LVI5KWZZFYPKQ, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.348469513 (574h46m5.219552558s) [noise=true , address=145.79.227.13:4804 , piece_id=LC2R4BSYDRRTJEZGYKHV32C67HVS6VQ6CFT6CATUOJLMPX2F73AA, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.348864155 (574h46m5.2199472s) [noise=true , address=145.79.227.46:4603 , piece_id=YLBY54LNV3O3NRKWIBFCTHDZVSXDH2M7AGVHRKSGOF6EKIEMC3OQ, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.349442666 (574h46m5.220525711s) [noise=true , address=145.79.227.6:4257 , piece_id=BOD75UC7DFIJ7H47EP72YN6LHRPMKLJLMXHFHKUPYH56BH7JCB6Q, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.350549544 (574h46m5.221632589s) [noise=true , address=145.79.227.17:4970 , piece_id=FR3NESBNJLG7F5DDCSYDNAASTKY2S3Z5GE4JODJ25X77DLBVZAKQ, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.365029656 (574h46m5.236112702s) [noise=true , address=145.79.227.39:4015 , piece_id=J4HS6HXM3ZYQGIL6JSYKFIDGUWRRX5C6HBSB2OWDDXILB7OJI77A, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.375620283 (574h46m5.246703328s) [noise=true , address=145.79.227.11:4729 , piece_id=4NZIKCEZ37GMZMQ2KUEQCFAQ3PBTH7S5R4LOGRSJUPGT7HVBUJWA, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.549604124 (574h46m5.420687159s) [noise=true , address=77.90.31.14:4663 , piece_id=3SHNXE7SWJKBHDTYFBLA2FYYBD3LLJ35DXRLQF2NX5EKDL4RTENA, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.607503303 (574h46m5.478586348s) [noise=true , address=77.90.31.16:4735 , piece_id=JZYNINGL7O6FRK2S324MDHAPRM2W3AUAAEHWLTBONFOHZ4CAGCXQ, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.643464388 (574h46m5.514547443s) [noise=true , address=45.91.116.7:4011 , piece_id=QSM5QVYA4NDTCXHWS42MIRYVYGYAUKS7F6DZELUKCWUOHPDU6CRA, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.648066741 (574h46m5.519149796s) [noise=true , address=45.91.116.11:4120 , piece_id=C5G7SX3HIBNIOS4F4VQWSAO6MLITBKSIJH5BPKWLVB5X4DXQN3ZQ, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.674957786 (574h46m5.546040831s) [noise=true , address=45.91.116.6:4974 , piece_id=53DHBHYME6SXPQU2YRA4VAKDTPEFLZNV62P4G6KQ54QQUBPIDYJQ, seg_pos={0 0}, upload=503874]: Done uploading piece. err: <nil> Dec 29 14:15:05.674986771 (574h46m5.546069816s) [seg_pos={0 0}, upload=503874]: Segment reached optimal threshold of 13 pieces. Dec 29 14:15:05.675327710 (574h46m5.546410755s) [noise=true , address=45.91.116.13:4153 , piece_id=N7OCYCLSMTGXC6JGXQBPILPTPMAO6H4GNLQ5BHOAQRNXUGROIY5Q, seg_pos={0 0}, upload=503874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 29 14:15:05.675338140 (574h46m5.546421175s) [noise=true , address=45.91.116.13:4153 , piece_id=N7OCYCLSMTGXC6JGXQBPILPTPMAO6H4GNLQ5BHOAQRNXUGROIY5Q, seg_pos={0 0}, upload=503874]: Long tail cancellation due to optimal threshold Dec 29 14:15:05.675640636 (574h46m5.546723691s) [noise=true , address=145.79.227.44:4195 , piece_id=T2GAKUTG3RERSAD6CTOVH6AWTNIQD2UNI5JFXXIB4W3YFSUDREFA, seg_pos={0 0}, upload=503874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 29 14:15:05.675658290 (574h46m5.546741335s) [noise=true , address=145.79.227.44:4195 , piece_id=T2GAKUTG3RERSAD6CTOVH6AWTNIQD2UNI5JFXXIB4W3YFSUDREFA, seg_pos={0 0}, upload=503874]: Long tail cancellation due to optimal threshold Dec 29 14:15:05.675694469 (574h46m5.546777614s) [noise=true , address=145.79.227.30:4761 , piece_id=VLQUQQK64ELKAVLZOCW6DIQVG4RHVUDCHQ665RWJY3UDZCLKNWRA, seg_pos={0 0}, upload=503874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 29 14:15:05.675706211 (574h46m5.546789256s) [noise=true , address=145.79.227.30:4761 , piece_id=VLQUQQK64ELKAVLZOCW6DIQVG4RHVUDCHQ665RWJY3UDZCLKNWRA, seg_pos={0 0}, upload=503874]: Long tail cancellation due to optimal threshold Dec 29 14:15:05.675728604 (574h46m5.546811649s) [seg_pos={0 0}, upload=503874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 14:15:05.675827642 (574h46m5.546910687s) [upload=503874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 14:15:05.675834495 (574h46m5.54691754s) [upload=503874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 14:15:05.964314143 (574h46m5.835397198s) [upload=503874]: Done uploading segments... Dec 29 14:15:05.964342036 (574h46m5.835425081s) [upload=503874]: Upload finished. err: <nil> Duration: 30.00374857s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration: 5.001077385s trace: https://jaeger.dev.storj.io/trace/6ff2a03f210af0c FAIL (2025-12-29 14:15:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00110054s core uplink us-select-1 us1-placement-20

1
...
11284
11285
11286
11287
11288
...
18076