Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
31.485201347s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:47:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 31.48521952s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...X.X...XXX.] duration:
5.844933782s trace:
https://jaeger.dev.storj.io/trace/39d3c6c039fcf272
FAIL (2025-12-31 18:46:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
gments...
Dec 31 18:46:11.888766278 (627h17m11.759849323s) [upload=549667]: Got next segment. Inline: false
Dec 31 18:46:11.888804371 (627h17m11.759887406s) [upload=549667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:46:11.888809941 (627h17m11.759892976s) [upload=549667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:46:12.138007799 (627h17m12.009090844s) [seg_pos={0 0}, upload=549667]: Begin upload segment...
Dec 31 18:46:12.138058545 (627h17m12.009141581s) [seg_pos={0 0}, upload=549667]: Done begin upload segment.
Dec 31 18:46:12.138100645 (627h17m12.00918369s) [noise=true , address=145.79.227.52:4447 , piece_id=TJSWMEYT7W7G4JSM7KB2STV5I3PGW5ZMPCJCYROWPXOO24MTY24A, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.138130753 (627h17m12.009213788s) [noise=true , address=145.79.227.6:4259 , piece_id=DCE5L36IRL6PZ6TTEBFOTBJBTP6KKIQKQVQFWCHOSYFMFSW7XBHQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.138942869 (627h17m12.010025914s) [noise=true , address=145.79.227.51:4411 , piece_id=6KBI2LASFRA7TU7IJD2OEYOLB2TSSWNM5FPKIL5TUJA3WVP372BQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.139442892 (627h17m12.010525927s) [noise=true , address=45.91.116.12:4228 , piece_id=K5RMXXI2SGQ6J23ECVQZNJYQDJLT7TNEMGD42T24I6NQKUPLDAIQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.139887729 (627h17m12.010970764s) [noise=true , address=77.90.31.11:4555 , piece_id=XK3YRDLIP2IWIDIS53P6S3ROKFTTZFN37NQEHEYTWSCC6CA2PP2Q, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.141408123 (627h17m12.012491158s) [noise=true , address=145.79.227.19:4079 , piece_id=QIA7PPEOYMJQFFN3UM7BCMJRAL2E7VFCQC5R3RZD2R5IKOLR3OYA, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.141921070 (627h17m12.013004115s) [noise=true , address=145.79.227.41:4085 , piece_id=NPXQSR3CCBKJ5R5DIH4LQAVGKBTTR3U6RWOSO7ENCLQUVAGXTBWQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.142781860 (627h17m12.013864895s) [noise=true , address=45.91.116.9:4261 , piece_id=IBWGMDFEC547KBO47D4ZURQJCMQCNXYTAJEUQYORRVIOZBZG2OIQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.144041749 (627h17m12.015124794s) [noise=true , address=77.90.31.14:4661 , piece_id=EJU6VGQGGEDKDGTPN76GEUEW66TB24MF5KT42N6YXVX6E7O7SYEQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.145305344 (627h17m12.016388379s) [noise=true , address=145.79.227.27:4691 , piece_id=4A3VIVD7NVFJP6ZYAIFSBHGECFLS42J2RRAFN4DSPMB6RQKYQI5Q, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.146473639 (627h17m12.017556675s) [noise=true , address=145.79.227.3:4150 , piece_id=MGYB437IHHI54GKL524BHW3DMIBELLLXIV5W54YMIQN3BZS5CGFA, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.147657864 (627h17m12.01874091s) [noise=true , address=145.79.227.40:4049 , piece_id=5INEVVXUM4CKJ6K6FXRRB5KN3ZZNVS6FGEQZK7IQEKZSHGFXQ5UA, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.147835302 (627h17m12.018918347s) [noise=true , address=45.91.116.11:4118 , piece_id=C7XXP446ZA3LT73T7XHVXNMPJJSVRBRQTUIGKNB6LYMT5D2M5KCQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.149259434 (627h17m12.020342469s) [noise=true , address=145.79.227.28:4727 , piece_id=RZ2YEXFAPUK33XAK3RXH5NIXKBVGKERQH7HSX5PIMMUO4EV4EBUA, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.149816224 (627h17m12.020899259s) [noise=true , address=145.79.227.46:4603 , piece_id=UYIYFACFL4UTBD2KVIMNIAW2HZQRJ764N4N53EDHEMJJSEQV6EAQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.150983797 (627h17m12.022066842s) [noise=true , address=145.79.227.5:4223 , piece_id=4QEEEGN6GTPW3J3L2JK473WTOSJBNJTAHO4MLSGT3B7QJSX6RC2Q, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:13.845223347 (627h17m13.716306392s) [upload=549667]: Next returned nil segment
Dec 31 18:46:13.845241601 (627h17m13.716324646s) [upload=549667]: Waiting for error group managing segments...
Dec 31 18:46:14.127164582 (627h17m13.998247617s) [noise=true , address=145.79.227.27:4691 , piece_id=4A3VIVD7NVFJP6ZYAIFSBHGECFLS42J2RRAFN4DSPMB6RQKYQI5Q, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.129292684 (627h17m14.000375719s) [noise=true , address=145.79.227.46:4603 , piece_id=UYIYFACFL4UTBD2KVIMNIAW2HZQRJ764N4N53EDHEMJJSEQV6EAQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.129907566 (627h17m14.000990611s) [noise=true , address=145.79.227.52:4447 , piece_id=TJSWMEYT7W7G4JSM7KB2STV5I3PGW5ZMPCJCYROWPXOO24MTY24A, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.131239191 (627h17m14.002322226s) [noise=true , address=145.79.227.6:4259 , piece_id=DCE5L36IRL6PZ6TTEBFOTBJBTP6KKIQKQVQFWCHOSYFMFSW7XBHQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.140415934 (627h17m14.01149898s) [noise=true , address=145.79.227.19:4079 , piece_id=QIA7PPEOYMJQFFN3UM7BCMJRAL2E7VFCQC5R3RZD2R5IKOLR3OYA, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.148946729 (627h17m14.020029774s) [noise=true , address=145.79.227.51:4411 , piece_id=6KBI2LASFRA7TU7IJD2OEYOLB2TSSWNM5FPKIL5TUJA3WVP372BQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.156379682 (627h17m14.027462727s) [noise=true , address=145.79.227.5:4223 , piece_id=4QEEEGN6GTPW3J3L2JK473WTOSJBNJTAHO4MLSGT3B7QJSX6RC2Q, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.158560755 (627h17m14.02964379s) [noise=true , address=145.79.227.41:4085 , piece_id=NPXQSR3CCBKJ5R5DIH4LQAVGKBTTR3U6RWOSO7ENCLQUVAGXTBWQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.159138956 (627h17m14.030222002s) [noise=true , address=145.79.227.28:4727 , piece_id=RZ2YEXFAPUK33XAK3RXH5NIXKBVGKERQH7HSX5PIMMUO4EV4EBUA, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.172081561 (627h17m14.043164606s) [noise=true , address=145.79.227.3:4150 , piece_id=MGYB437IHHI54GKL524BHW3DMIBELLLXIV5W54YMIQN3BZS5CGFA, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.313933516 (627h17m14.185016551s) [noise=true , address=77.90.31.11:4555 , piece_id=XK3YRDLIP2IWIDIS53P6S3ROKFTTZFN37NQEHEYTWSCC6CA2PP2Q, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.441196343 (627h17m14.312279378s) [noise=true , address=77.90.31.14:4661 , piece_id=EJU6VGQGGEDKDGTPN76GEUEW66TB24MF5KT42N6YXVX6E7O7SYEQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.486714596 (627h17m14.357797631s) [noise=true , address=45.91.116.11:4118 , piece_id=C7XXP446ZA3LT73T7XHVXNMPJJSVRBRQTUIGKNB6LYMT5D2M5KCQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.486735826 (627h17m14.357818871s) [seg_pos={0 0}, upload=549667]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:46:14.486944383 (627h17m14.358027418s) [noise=true , address=145.79.227.40:4049 , piece_id=5INEVVXUM4CKJ6K6FXRRB5KN3ZZNVS6FGEQZK7IQEKZSHGFXQ5UA, seg_pos={0 0}, upload=549667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 31 18:46:14.486950716 (627h17m14.358033751s) [noise=true , address=145.79.227.40:4049 , piece_id=5INEVVXUM4CKJ6K6FXRRB5KN3ZZNVS6FGEQZK7IQEKZSHGFXQ5UA, seg_pos={0 0}, upload=549667]: Long tail cancellation due to optimal threshold
Dec 31 18:46:14.487051326 (627h17m14.358134371s) [noise=true , address=45.91.116.12:4228 , piece_id=K5RMXXI2SGQ6J23ECVQZNJYQDJLT7TNEMGD42T24I6NQKUPLDAIQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:42018->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:42018->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42018->45.91.116.12:4228: use of closed network connection
Dec 31 18:46:14.487072106 (627h17m14.358155151s) [noise=true , address=45.91.116.12:4228 , piece_id=K5RMXXI2SGQ6J23ECVQZNJYQDJLT7TNEMGD42T24I6NQKUPLDAIQ, seg_pos={0 0}, upload=549667]: Long tail cancellation due to optimal threshold
Dec 31 18:46:14.487175223 (627h17m14.358258278s) [noise=true , address=45.91.116.9:4261 , piece_id=IBWGMDFEC547KBO47D4ZURQJCMQCNXYTAJEUQYORRVIOZBZG2OIQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:36728->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:36728->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36728->45.91.116.9:4261: use of closed network connection
Dec 31 18:46:14.487185312 (627h17m14.358268367s) [noise=true , address=45.91.116.9:4261 , piece_id=IBWGMDFEC547KBO47D4ZURQJCMQCNXYTAJEUQYORRVIOZBZG2OIQ, seg_pos={0 0}, upload=549667]: Long tail cancellation due to optimal threshold
Dec 31 18:46:14.487211221 (627h17m14.358294266s) [seg_pos={0 0}, upload=549667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:46:14.487308096 (627h17m14.358391141s) [upload=549667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:46:14.487314187 (627h17m14.358397222s) [upload=549667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:46:14.789329618 (627h17m14.660412663s) [upload=549667]: Done uploading segments...
Dec 31 18:46:14.789367069 (627h17m14.660450114s) [upload=549667]: Upload finished. err: <nil>
Duration: 5.844951025s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...X.X...XXX] duration:
30.002096628s trace:
https://jaeger.dev.storj.io/trace/39d3c6c039fcf272
FAIL (2025-12-31 18:46:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
gments...
Dec 31 18:46:11.888766278 (627h17m11.759849323s) [upload=549667]: Got next segment. Inline: false
Dec 31 18:46:11.888804371 (627h17m11.759887406s) [upload=549667]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:46:11.888809941 (627h17m11.759892976s) [upload=549667]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:46:12.138007799 (627h17m12.009090844s) [seg_pos={0 0}, upload=549667]: Begin upload segment...
Dec 31 18:46:12.138058545 (627h17m12.009141581s) [seg_pos={0 0}, upload=549667]: Done begin upload segment.
Dec 31 18:46:12.138100645 (627h17m12.00918369s) [noise=true , address=145.79.227.52:4447 , piece_id=TJSWMEYT7W7G4JSM7KB2STV5I3PGW5ZMPCJCYROWPXOO24MTY24A, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.138130753 (627h17m12.009213788s) [noise=true , address=145.79.227.6:4259 , piece_id=DCE5L36IRL6PZ6TTEBFOTBJBTP6KKIQKQVQFWCHOSYFMFSW7XBHQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.138942869 (627h17m12.010025914s) [noise=true , address=145.79.227.51:4411 , piece_id=6KBI2LASFRA7TU7IJD2OEYOLB2TSSWNM5FPKIL5TUJA3WVP372BQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.139442892 (627h17m12.010525927s) [noise=true , address=45.91.116.12:4228 , piece_id=K5RMXXI2SGQ6J23ECVQZNJYQDJLT7TNEMGD42T24I6NQKUPLDAIQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.139887729 (627h17m12.010970764s) [noise=true , address=77.90.31.11:4555 , piece_id=XK3YRDLIP2IWIDIS53P6S3ROKFTTZFN37NQEHEYTWSCC6CA2PP2Q, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.141408123 (627h17m12.012491158s) [noise=true , address=145.79.227.19:4079 , piece_id=QIA7PPEOYMJQFFN3UM7BCMJRAL2E7VFCQC5R3RZD2R5IKOLR3OYA, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.141921070 (627h17m12.013004115s) [noise=true , address=145.79.227.41:4085 , piece_id=NPXQSR3CCBKJ5R5DIH4LQAVGKBTTR3U6RWOSO7ENCLQUVAGXTBWQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.142781860 (627h17m12.013864895s) [noise=true , address=45.91.116.9:4261 , piece_id=IBWGMDFEC547KBO47D4ZURQJCMQCNXYTAJEUQYORRVIOZBZG2OIQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.144041749 (627h17m12.015124794s) [noise=true , address=77.90.31.14:4661 , piece_id=EJU6VGQGGEDKDGTPN76GEUEW66TB24MF5KT42N6YXVX6E7O7SYEQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.145305344 (627h17m12.016388379s) [noise=true , address=145.79.227.27:4691 , piece_id=4A3VIVD7NVFJP6ZYAIFSBHGECFLS42J2RRAFN4DSPMB6RQKYQI5Q, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.146473639 (627h17m12.017556675s) [noise=true , address=145.79.227.3:4150 , piece_id=MGYB437IHHI54GKL524BHW3DMIBELLLXIV5W54YMIQN3BZS5CGFA, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.147657864 (627h17m12.01874091s) [noise=true , address=145.79.227.40:4049 , piece_id=5INEVVXUM4CKJ6K6FXRRB5KN3ZZNVS6FGEQZK7IQEKZSHGFXQ5UA, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.147835302 (627h17m12.018918347s) [noise=true , address=45.91.116.11:4118 , piece_id=C7XXP446ZA3LT73T7XHVXNMPJJSVRBRQTUIGKNB6LYMT5D2M5KCQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.149259434 (627h17m12.020342469s) [noise=true , address=145.79.227.28:4727 , piece_id=RZ2YEXFAPUK33XAK3RXH5NIXKBVGKERQH7HSX5PIMMUO4EV4EBUA, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.149816224 (627h17m12.020899259s) [noise=true , address=145.79.227.46:4603 , piece_id=UYIYFACFL4UTBD2KVIMNIAW2HZQRJ764N4N53EDHEMJJSEQV6EAQ, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:12.150983797 (627h17m12.022066842s) [noise=true , address=145.79.227.5:4223 , piece_id=4QEEEGN6GTPW3J3L2JK473WTOSJBNJTAHO4MLSGT3B7QJSX6RC2Q, seg_pos={0 0}, upload=549667]: Uploading piece...
Dec 31 18:46:13.845223347 (627h17m13.716306392s) [upload=549667]: Next returned nil segment
Dec 31 18:46:13.845241601 (627h17m13.716324646s) [upload=549667]: Waiting for error group managing segments...
Dec 31 18:46:14.127164582 (627h17m13.998247617s) [noise=true , address=145.79.227.27:4691 , piece_id=4A3VIVD7NVFJP6ZYAIFSBHGECFLS42J2RRAFN4DSPMB6RQKYQI5Q, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.129292684 (627h17m14.000375719s) [noise=true , address=145.79.227.46:4603 , piece_id=UYIYFACFL4UTBD2KVIMNIAW2HZQRJ764N4N53EDHEMJJSEQV6EAQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.129907566 (627h17m14.000990611s) [noise=true , address=145.79.227.52:4447 , piece_id=TJSWMEYT7W7G4JSM7KB2STV5I3PGW5ZMPCJCYROWPXOO24MTY24A, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.131239191 (627h17m14.002322226s) [noise=true , address=145.79.227.6:4259 , piece_id=DCE5L36IRL6PZ6TTEBFOTBJBTP6KKIQKQVQFWCHOSYFMFSW7XBHQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.140415934 (627h17m14.01149898s) [noise=true , address=145.79.227.19:4079 , piece_id=QIA7PPEOYMJQFFN3UM7BCMJRAL2E7VFCQC5R3RZD2R5IKOLR3OYA, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.148946729 (627h17m14.020029774s) [noise=true , address=145.79.227.51:4411 , piece_id=6KBI2LASFRA7TU7IJD2OEYOLB2TSSWNM5FPKIL5TUJA3WVP372BQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.156379682 (627h17m14.027462727s) [noise=true , address=145.79.227.5:4223 , piece_id=4QEEEGN6GTPW3J3L2JK473WTOSJBNJTAHO4MLSGT3B7QJSX6RC2Q, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.158560755 (627h17m14.02964379s) [noise=true , address=145.79.227.41:4085 , piece_id=NPXQSR3CCBKJ5R5DIH4LQAVGKBTTR3U6RWOSO7ENCLQUVAGXTBWQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.159138956 (627h17m14.030222002s) [noise=true , address=145.79.227.28:4727 , piece_id=RZ2YEXFAPUK33XAK3RXH5NIXKBVGKERQH7HSX5PIMMUO4EV4EBUA, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.172081561 (627h17m14.043164606s) [noise=true , address=145.79.227.3:4150 , piece_id=MGYB437IHHI54GKL524BHW3DMIBELLLXIV5W54YMIQN3BZS5CGFA, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.313933516 (627h17m14.185016551s) [noise=true , address=77.90.31.11:4555 , piece_id=XK3YRDLIP2IWIDIS53P6S3ROKFTTZFN37NQEHEYTWSCC6CA2PP2Q, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.441196343 (627h17m14.312279378s) [noise=true , address=77.90.31.14:4661 , piece_id=EJU6VGQGGEDKDGTPN76GEUEW66TB24MF5KT42N6YXVX6E7O7SYEQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.486714596 (627h17m14.357797631s) [noise=true , address=45.91.116.11:4118 , piece_id=C7XXP446ZA3LT73T7XHVXNMPJJSVRBRQTUIGKNB6LYMT5D2M5KCQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: <nil>
Dec 31 18:46:14.486735826 (627h17m14.357818871s) [seg_pos={0 0}, upload=549667]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:46:14.486944383 (627h17m14.358027418s) [noise=true , address=145.79.227.40:4049 , piece_id=5INEVVXUM4CKJ6K6FXRRB5KN3ZZNVS6FGEQZK7IQEKZSHGFXQ5UA, seg_pos={0 0}, upload=549667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 31 18:46:14.486950716 (627h17m14.358033751s) [noise=true , address=145.79.227.40:4049 , piece_id=5INEVVXUM4CKJ6K6FXRRB5KN3ZZNVS6FGEQZK7IQEKZSHGFXQ5UA, seg_pos={0 0}, upload=549667]: Long tail cancellation due to optimal threshold
Dec 31 18:46:14.487051326 (627h17m14.358134371s) [noise=true , address=45.91.116.12:4228 , piece_id=K5RMXXI2SGQ6J23ECVQZNJYQDJLT7TNEMGD42T24I6NQKUPLDAIQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:42018->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:42018->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42018->45.91.116.12:4228: use of closed network connection
Dec 31 18:46:14.487072106 (627h17m14.358155151s) [noise=true , address=45.91.116.12:4228 , piece_id=K5RMXXI2SGQ6J23ECVQZNJYQDJLT7TNEMGD42T24I6NQKUPLDAIQ, seg_pos={0 0}, upload=549667]: Long tail cancellation due to optimal threshold
Dec 31 18:46:14.487175223 (627h17m14.358258278s) [noise=true , address=45.91.116.9:4261 , piece_id=IBWGMDFEC547KBO47D4ZURQJCMQCNXYTAJEUQYORRVIOZBZG2OIQ, seg_pos={0 0}, upload=549667]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:36728->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:36728->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36728->45.91.116.9:4261: use of closed network connection
Dec 31 18:46:14.487185312 (627h17m14.358268367s) [noise=true , address=45.91.116.9:4261 , piece_id=IBWGMDFEC547KBO47D4ZURQJCMQCNXYTAJEUQYORRVIOZBZG2OIQ, seg_pos={0 0}, upload=549667]: Long tail cancellation due to optimal threshold
Dec 31 18:46:14.487211221 (627h17m14.358294266s) [seg_pos={0 0}, upload=549667]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:46:14.487308096 (627h17m14.358391141s) [upload=549667]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:46:14.487314187 (627h17m14.358397222s) [upload=549667]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:46:14.789329618 (627h17m14.660412663s) [upload=549667]: Done uploading segments...
Dec 31 18:46:14.789367069 (627h17m14.660450114s) [upload=549667]: Upload finished. err: <nil>
Duration: 30.002129269s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.197065708s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:46:30 UTC)
orbiter warning: context deadline exceeded
Duration: 6.197093401s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
24.241100218s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:46:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 24.241127582s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.X...X.X...................X................................] duration:
3.597397021s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:46:22 UTC)
orbiter warning: context deadline exceeded
Duration: 3.597424413s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
5.924414535s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:45:59 UTC)
orbiter warning: context deadline exceeded
Duration: 5.924434152s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
29.417247549s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:45:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 29.417278307s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
6.17505097s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:45:38 UTC)
orbiter warning: context deadline exceeded
Duration: 6.175079575s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...X.X...XX] duration:
30.001770981s trace:
https://jaeger.dev.storj.io/trace/3bd179aac3fc54a8
FAIL (2025-12-31 18:45:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, Piece ID: AGSGNGX62LV4I3HYO6JT2QRYQRCPTKAJW3EIGPU53JTJTGTLRBRA) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AYDQG5J4KTQQEW2MOCHI7PDGQQ4XKXQ6TVHMZ5ZIJWMOZMBSG26A) context deadline exceeded
======LOGS======
Dec 31 18:44:57.824104730 (627h15m57.695187765s) [upload=549651]: Starting upload
Dec 31 18:44:57.824158333 (627h15m57.695241368s) [upload=549651]: Done starting upload
Dec 31 18:44:57.824193400 (627h15m57.695276435s) [upload=549651]: Uploading segments...
Dec 31 18:44:57.824345669 (627h15m57.695428704s) [upload=549651]: Got next segment. Inline: false
Dec 31 18:44:57.824371980 (627h15m57.695455015s) [upload=549651]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:44:57.824376608 (627h15m57.695459643s) [upload=549651]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:44:58.074265341 (627h15m57.945348376s) [seg_pos={0 0}, upload=549651]: Begin upload segment...
Dec 31 18:44:58.074311490 (627h15m57.945394535s) [seg_pos={0 0}, upload=549651]: Done begin upload segment.
Dec 31 18:44:58.074345144 (627h15m57.945428209s) [noise=true , address=45.91.116.11:4119 , piece_id=AZSITQNRV54FFB6OMTW65JQYQVN65NETKHNKL66A4CCXBTUSHKBA, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.074396381 (627h15m57.945479426s) [noise=true , address=145.79.227.43:4158 , piece_id=IYGFNQQUIJRBQPS2ZH6RB4MTCQAPAPC4AJCG7PFQRZC2S7X4NATQ, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.074875754 (627h15m57.945958799s) [noise=true , address=77.90.31.14:4661 , piece_id=OKNDYP6E3YNWEIKWL3B5KZT5N2ISO36DJZEQPJJTJEDXRTP6IIYA, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.075204570 (627h15m57.946287605s) [noise=true , address=145.79.227.24:4583 , piece_id=MZ66ZYFZWK2R3MFYK3YT26MHOILC5DDKU4AVMSEPNCETJP6WAE5A, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.075540890 (627h15m57.946623925s) [noise=true , address=145.79.227.8:4331 , piece_id=RB76BX33Z5GDOBBJI5JCMLEXJCKHVSMTNWGAYZUA2TEPYCZYLJ6Q, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.076117519 (627h15m57.947200554s) [noise=true , address=145.79.227.13:4802 , piece_id=EZTESNWTTRUGW4SSHHUY6I5AAYDPCJ7LEEJNXG36WMLJ4VJX2HZA, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.076263306 (627h15m57.947346341s) [noise=true , address=145.79.227.27:4689 , piece_id=32QXTHP3MUXPUSEKOHAGISCKWYKA4TH63DLND6GZ64NEVH3SM2CQ, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.076551374 (627h15m57.94763442s) [noise=true , address=145.79.227.36:4517 , piece_id=KCM5XX3DVQDD5UNGHMJVPU3WKS7YXUNQMWG72RRKZSXQJ26IFDFA, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.076846908 (627h15m57.947929943s) [noise=true , address=77.90.31.17:4773 , piece_id=AGSGNGX62LV4I3HYO6JT2QRYQRCPTKAJW3EIGPU53JTJTGTLRBRA, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.077273700 (627h15m57.948356745s) [noise=true , address=145.79.227.48:4303 , piece_id=252A2W5M5RQYZMLSFTOCQZWQSFMCVRMLBYVGXGNHJF2RPTOQVKRQ, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.077333354 (627h15m57.948416389s) [noise=true , address=145.79.227.47:4267 , piece_id=OIPMYO6ALYIVCGRHMONX4O7BYHJWID5HA6AIS5HXHEEHNS3UALWQ, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.077630599 (627h15m57.948713635s) [noise=true , address=145.79.227.41:4085 , piece_id=O2TOCIFJPWIPRI56WZ6DNGILHPNIPPWN2KWMMX56VI7W3RLZGJ3A, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.077675094 (627h15m57.948758139s) [noise=true , address=77.90.31.15:4697 , piece_id=53QUTEO6LIDD6QZDY74J2GML7WKPTLM5FT6E3DIRUM3FPFGP4EBA, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.077978582 (627h15m57.949061617s) [noise=true , address=45.91.116.13:4153 , piece_id=AYDQG5J4KTQQEW2MOCHI7PDGQQ4XKXQ6TVHMZ5ZIJWMOZMBSG26A, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.078154447 (627h15m57.949237482s) [noise=true , address=145.79.227.2:4115 , piece_id=3OZ3XCE6BP6JXD6IHJ6NO3GPNHN24EFOJAQW6GK7VYT7UUZGIMQA, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:58.078261421 (627h15m57.949344456s) [noise=true , address=145.79.227.39:4014 , piece_id=QPYHYMQEAXF3QJKUJON2HTCUYDLZUU4B5M2JIDIUTYUPV4KFHAWQ, seg_pos={0 0}, upload=549651]: Uploading piece...
Dec 31 18:44:59.776933532 (627h15m59.648016567s) [upload=549651]: Next returned nil segment
Dec 31 18:44:59.776953750 (627h15m59.648036785s) [upload=549651]: Waiting for error group managing segments...
Dec 31 18:45:00.052334764 (627h15m59.923417799s) [noise=true , address=145.79.227.48:4303 , piece_id=252A2W5M5RQYZMLSFTOCQZWQSFMCVRMLBYVGXGNHJF2RPTOQVKRQ, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.052462398 (627h15m59.923545443s) [noise=true , address=145.79.227.13:4802 , piece_id=EZTESNWTTRUGW4SSHHUY6I5AAYDPCJ7LEEJNXG36WMLJ4VJX2HZA, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.066974771 (627h15m59.938057806s) [noise=true , address=145.79.227.36:4517 , piece_id=KCM5XX3DVQDD5UNGHMJVPU3WKS7YXUNQMWG72RRKZSXQJ26IFDFA, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.072911496 (627h15m59.943994541s) [noise=true , address=145.79.227.47:4267 , piece_id=OIPMYO6ALYIVCGRHMONX4O7BYHJWID5HA6AIS5HXHEEHNS3UALWQ, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.073338519 (627h15m59.944421564s) [noise=true , address=145.79.227.39:4014 , piece_id=QPYHYMQEAXF3QJKUJON2HTCUYDLZUU4B5M2JIDIUTYUPV4KFHAWQ, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.081883730 (627h15m59.952966775s) [noise=true , address=145.79.227.2:4115 , piece_id=3OZ3XCE6BP6JXD6IHJ6NO3GPNHN24EFOJAQW6GK7VYT7UUZGIMQA, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.091033904 (627h15m59.962116939s) [noise=true , address=145.79.227.8:4331 , piece_id=RB76BX33Z5GDOBBJI5JCMLEXJCKHVSMTNWGAYZUA2TEPYCZYLJ6Q, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.091069521 (627h15m59.962152556s) [noise=true , address=145.79.227.41:4085 , piece_id=O2TOCIFJPWIPRI56WZ6DNGILHPNIPPWN2KWMMX56VI7W3RLZGJ3A, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.091477918 (627h15m59.962560953s) [noise=true , address=145.79.227.24:4583 , piece_id=MZ66ZYFZWK2R3MFYK3YT26MHOILC5DDKU4AVMSEPNCETJP6WAE5A, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.104744610 (627h15m59.975827655s) [noise=true , address=145.79.227.27:4689 , piece_id=32QXTHP3MUXPUSEKOHAGISCKWYKA4TH63DLND6GZ64NEVH3SM2CQ, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.248399659 (627h16m0.119482725s) [noise=true , address=77.90.31.17:4773 , piece_id=AGSGNGX62LV4I3HYO6JT2QRYQRCPTKAJW3EIGPU53JTJTGTLRBRA, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.309674917 (627h16m0.180757962s) [noise=true , address=77.90.31.15:4697 , piece_id=53QUTEO6LIDD6QZDY74J2GML7WKPTLM5FT6E3DIRUM3FPFGP4EBA, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.401313790 (627h16m0.272396825s) [noise=true , address=45.91.116.13:4153 , piece_id=AYDQG5J4KTQQEW2MOCHI7PDGQQ4XKXQ6TVHMZ5ZIJWMOZMBSG26A, seg_pos={0 0}, upload=549651]: Done uploading piece. err: <nil>
Dec 31 18:45:00.401332004 (627h16m0.272415049s) [seg_pos={0 0}, upload=549651]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:45:00.401502609 (627h16m0.272585654s) [noise=true , address=77.90.31.14:4661 , piece_id=OKNDYP6E3YNWEIKWL3B5KZT5N2ISO36DJZEQPJJTJEDXRTP6IIYA, seg_pos={0 0}, upload=549651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:41370->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:41370->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41370->77.90.31.14:4661: use of closed network connection
Dec 31 18:45:00.401508571 (627h16m0.272591616s) [noise=true , address=77.90.31.14:4661 , piece_id=OKNDYP6E3YNWEIKWL3B5KZT5N2ISO36DJZEQPJJTJEDXRTP6IIYA, seg_pos={0 0}, upload=549651]: Long tail cancellation due to optimal threshold
Dec 31 18:45:00.401560569 (627h16m0.272643615s) [noise=true , address=145.79.227.43:4158 , piece_id=IYGFNQQUIJRBQPS2ZH6RB4MTCQAPAPC4AJCG7PFQRZC2S7X4NATQ, seg_pos={0 0}, upload=549651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 31 18:45:00.401569055 (627h16m0.2726521s) [noise=true , address=145.79.227.43:4158 , piece_id=IYGFNQQUIJRBQPS2ZH6RB4MTCQAPAPC4AJCG7PFQRZC2S7X4NATQ, seg_pos={0 0}, upload=549651]: Long tail cancellation due to optimal threshold
Dec 31 18:45:00.401671020 (627h16m0.272754075s) [noise=true , address=45.91.116.11:4119 , piece_id=AZSITQNRV54FFB6OMTW65JQYQVN65NETKHNKL66A4CCXBTUSHKBA, seg_pos={0 0}, upload=549651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 31 18:45:00.401680067 (627h16m0.272763112s) [noise=true , address=45.91.116.11:4119 , piece_id=AZSITQNRV54FFB6OMTW65JQYQVN65NETKHNKL66A4CCXBTUSHKBA, seg_pos={0 0}, upload=549651]: Long tail cancellation due to optimal threshold
Dec 31 18:45:00.401763526 (627h16m0.272846561s) [seg_pos={0 0}, upload=549651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:45:00.401884746 (627h16m0.272967792s) [upload=549651]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:45:00.401891129 (627h16m0.272974164s) [upload=549651]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:45:00.692700261 (627h16m0.563783326s) [upload=549651]: Done uploading segments...
Dec 31 18:45:00.692724917 (627h16m0.563807963s) [upload=549651]: Upload finished. err: <nil>
Duration: 30.001792321s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..X...X.X...................X...............................] duration:
3.405497488s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:45:17 UTC)
orbiter warning: context deadline exceeded
Duration: 3.405521473s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
6.242309357s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:45:03 UTC)
orbiter warning: context deadline exceeded
Duration: 6.242341247s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
43.776062266s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:44:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 43.776074619s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [..X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...X.X...X] duration:
30.002835154s trace:
https://jaeger.dev.storj.io/trace/27990027439def42
FAIL (2025-12-31 18:44:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: NWL7TPEHYORMPREMHTGFE6XLGVILCNRC6S3P3DUPXZBREOGRN2IQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: EH2Z62PD3CRZNLVQUNE726UIB7A6LEBJPRKVYIA243PG6E344HCA) context deadline exceeded
======LOGS======
31 18:44:12.535318547 (627h15m12.406401592s) [upload=549639]: Starting upload
Dec 31 18:44:12.535373912 (627h15m12.406456947s) [upload=549639]: Done starting upload
Dec 31 18:44:12.535475826 (627h15m12.406558891s) [upload=549639]: Uploading segments...
Dec 31 18:44:12.535579864 (627h15m12.406662909s) [upload=549639]: Got next segment. Inline: false
Dec 31 18:44:12.535615722 (627h15m12.406698767s) [upload=549639]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:44:12.535621834 (627h15m12.406704879s) [upload=549639]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:44:12.781851329 (627h15m12.652934364s) [seg_pos={0 0}, upload=549639]: Begin upload segment...
Dec 31 18:44:12.781904430 (627h15m12.652987475s) [seg_pos={0 0}, upload=549639]: Done begin upload segment.
Dec 31 18:44:12.781937263 (627h15m12.653020298s) [noise=true , address=145.79.227.37:4567 , piece_id=MNGZ7YMZRF45LUVJPYMPLIXFSYXNHMUNNPXMZAVIHU3HHFJNTCXA, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.781951940 (627h15m12.653034985s) [noise=true , address=145.79.227.10:4402 , piece_id=2EAGJEM44W47CIZOD5XW2XJH26YQMBJRTSF673LOBOQVJHAUAGTA, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.782477511 (627h15m12.653560566s) [noise=true , address=145.79.227.38:4979 , piece_id=NLIEYA3L4EGYNWVAZCWFVPKIAXJEELMPF42WBHHBHPIDGIWJSGWQ, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.782909164 (627h15m12.653992209s) [noise=true , address=145.79.227.42:4123 , piece_id=XVFNWP7N2D7AEGRVH2KVSZWMCJHRBJCABT2JDHC4S3COKXGOQQ4A, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.783327069 (627h15m12.654410114s) [noise=true , address=145.79.227.15:4876 , piece_id=H4OD3SNWPC2NC6YWUP2KON6B2KMUWW2LU2AM2GEZ6J5Z3G6VKNPQ, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.783768810 (627h15m12.654851855s) [noise=true , address=145.79.227.2:4114 , piece_id=4BO4CSXTAH5LB6B77WNBQPA7QNDX5KWJZ6WINDBSWNYEKSQU3RDQ, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.784231221 (627h15m12.655314266s) [noise=true , address=145.79.227.19:4077 , piece_id=4XGCW2O2NK3AKUOPSVQCP7M2EYJG5PRTN2KCTK5UF56ZFUNLRY5Q, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.785348769 (627h15m12.656431804s) [noise=true , address=45.91.116.13:4154 , piece_id=WMKU7P7JKAUGKUDB47HZWUK74O7YLY2GGL36HYOKFCLIUNYPW22Q, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.785988156 (627h15m12.657071201s) [noise=true , address=77.90.31.11:4556 , piece_id=EFCWZFALIWQEWMB6VV7BRB73F2HKB4LVHNKMAYOSI33DJM26RHFA, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.786493399 (627h15m12.657576444s) [noise=true , address=77.90.31.21:4878 , piece_id=NWL7TPEHYORMPREMHTGFE6XLGVILCNRC6S3P3DUPXZBREOGRN2IQ, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.787496699 (627h15m12.658579734s) [noise=true , address=145.79.227.22:4509 , piece_id=QYC6BC7JI7RTV5OSAPANRZ32SM4HBELGV2IYOKABLFN7ZEDVKNUA, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.788620028 (627h15m12.659703063s) [noise=true , address=145.79.227.33:4870 , piece_id=JJT5LV34J7X2KEYZ6CWDAHHFJY3HUSMNDRRQQ7LQMO3F53S2AXIA, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.790348899 (627h15m12.661431934s) [noise=true , address=145.79.227.14:4839 , piece_id=XKOPRVADTB3X7ANIMQEBBNQQRBFZXIHMNOBNJ2FRPUBYMSMPNCZA, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.791832505 (627h15m12.66291555s) [noise=true , address=77.90.31.10:4519 , piece_id=EH2Z62PD3CRZNLVQUNE726UIB7A6LEBJPRKVYIA243PG6E344HCA, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.792994828 (627h15m12.664077863s) [noise=true , address=45.91.116.5:4298 , piece_id=Y6RHNQMCBBSLBT2F63BBSLZIFR74ECTDGUM4Q3SYGG7HVZ2XIEAQ, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:12.794105533 (627h15m12.665188578s) [noise=true , address=45.91.116.10:4083 , piece_id=G5E4EE2ICZBCEPRFWEM56UUNPGSPV73ZXAYHVXHTLIH4RIQ5NSXA, seg_pos={0 0}, upload=549639]: Uploading piece...
Dec 31 18:44:14.385277316 (627h15m14.256360362s) [upload=549639]: Next returned nil segment
Dec 31 18:44:14.385306773 (627h15m14.256389808s) [upload=549639]: Waiting for error group managing segments...
Dec 31 18:44:14.764988159 (627h15m14.636071194s) [noise=true , address=145.79.227.2:4114 , piece_id=4BO4CSXTAH5LB6B77WNBQPA7QNDX5KWJZ6WINDBSWNYEKSQU3RDQ, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:14.768034440 (627h15m14.639117475s) [noise=true , address=145.79.227.22:4509 , piece_id=QYC6BC7JI7RTV5OSAPANRZ32SM4HBELGV2IYOKABLFN7ZEDVKNUA, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:14.778537118 (627h15m14.649620153s) [noise=true , address=145.79.227.10:4402 , piece_id=2EAGJEM44W47CIZOD5XW2XJH26YQMBJRTSF673LOBOQVJHAUAGTA, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:14.780573195 (627h15m14.651656241s) [noise=true , address=145.79.227.15:4876 , piece_id=H4OD3SNWPC2NC6YWUP2KON6B2KMUWW2LU2AM2GEZ6J5Z3G6VKNPQ, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:14.782272251 (627h15m14.653355296s) [noise=true , address=145.79.227.42:4123 , piece_id=XVFNWP7N2D7AEGRVH2KVSZWMCJHRBJCABT2JDHC4S3COKXGOQQ4A, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:14.797885932 (627h15m14.668968967s) [noise=true , address=145.79.227.37:4567 , piece_id=MNGZ7YMZRF45LUVJPYMPLIXFSYXNHMUNNPXMZAVIHU3HHFJNTCXA, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:14.798801024 (627h15m14.669884059s) [noise=true , address=145.79.227.38:4979 , piece_id=NLIEYA3L4EGYNWVAZCWFVPKIAXJEELMPF42WBHHBHPIDGIWJSGWQ, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:14.807220675 (627h15m14.67830372s) [noise=true , address=145.79.227.14:4839 , piece_id=XKOPRVADTB3X7ANIMQEBBNQQRBFZXIHMNOBNJ2FRPUBYMSMPNCZA, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:14.854294952 (627h15m14.725377997s) [noise=true , address=145.79.227.19:4077 , piece_id=4XGCW2O2NK3AKUOPSVQCP7M2EYJG5PRTN2KCTK5UF56ZFUNLRY5Q, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:14.944326494 (627h15m14.815409529s) [noise=true , address=77.90.31.10:4519 , piece_id=EH2Z62PD3CRZNLVQUNE726UIB7A6LEBJPRKVYIA243PG6E344HCA, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:14.967135365 (627h15m14.83821841s) [noise=true , address=77.90.31.21:4878 , piece_id=NWL7TPEHYORMPREMHTGFE6XLGVILCNRC6S3P3DUPXZBREOGRN2IQ, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:15.094327366 (627h15m14.965410402s) [noise=true , address=77.90.31.11:4556 , piece_id=EFCWZFALIWQEWMB6VV7BRB73F2HKB4LVHNKMAYOSI33DJM26RHFA, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:15.114452358 (627h15m14.985535393s) [noise=true , address=45.91.116.10:4083 , piece_id=G5E4EE2ICZBCEPRFWEM56UUNPGSPV73ZXAYHVXHTLIH4RIQ5NSXA, seg_pos={0 0}, upload=549639]: Done uploading piece. err: <nil>
Dec 31 18:44:15.114472886 (627h15m14.985555921s) [seg_pos={0 0}, upload=549639]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:44:15.114648871 (627h15m14.985731906s) [noise=true , address=45.91.116.5:4298 , piece_id=Y6RHNQMCBBSLBT2F63BBSLZIFR74ECTDGUM4Q3SYGG7HVZ2XIEAQ, seg_pos={0 0}, upload=549639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:42142->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:42142->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42142->45.91.116.5:4298: use of closed network connection
Dec 31 18:44:15.114654572 (627h15m14.985737607s) [noise=true , address=45.91.116.5:4298 , piece_id=Y6RHNQMCBBSLBT2F63BBSLZIFR74ECTDGUM4Q3SYGG7HVZ2XIEAQ, seg_pos={0 0}, upload=549639]: Long tail cancellation due to optimal threshold
Dec 31 18:44:15.114837241 (627h15m14.985920286s) [noise=true , address=145.79.227.33:4870 , piece_id=JJT5LV34J7X2KEYZ6CWDAHHFJY3HUSMNDRRQQ7LQMO3F53S2AXIA, seg_pos={0 0}, upload=549639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 31 18:44:15.114842981 (627h15m14.985926016s) [noise=true , address=145.79.227.33:4870 , piece_id=JJT5LV34J7X2KEYZ6CWDAHHFJY3HUSMNDRRQQ7LQMO3F53S2AXIA, seg_pos={0 0}, upload=549639]: Long tail cancellation due to optimal threshold
Dec 31 18:44:15.114997555 (627h15m14.98608061s) [noise=true , address=45.91.116.13:4154 , piece_id=WMKU7P7JKAUGKUDB47HZWUK74O7YLY2GGL36HYOKFCLIUNYPW22Q, seg_pos={0 0}, upload=549639]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 18:44:15.115036179 (627h15m14.986119224s) [noise=true , address=45.91.116.13:4154 , piece_id=WMKU7P7JKAUGKUDB47HZWUK74O7YLY2GGL36HYOKFCLIUNYPW22Q, seg_pos={0 0}, upload=549639]: Long tail cancellation due to optimal threshold
Dec 31 18:44:15.115063872 (627h15m14.986146917s) [seg_pos={0 0}, upload=549639]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:44:15.115219098 (627h15m14.986302143s) [upload=549639]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:44:15.115227033 (627h15m14.986310078s) [upload=549639]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:44:15.416146345 (627h15m15.28722938s) [upload=549639]: Done uploading segments...
Dec 31 18:44:15.416170090 (627h15m15.287253135s) [upload=549639]: Upload finished. err: <nil>
Duration: 30.002860403s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
36.359331756s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:44:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 36.359349219s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...X...X.X...................X..............................] duration:
3.519902183s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:44:10 UTC)
orbiter warning: context deadline exceeded
Duration: 3.519933081s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
14.846842817s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:44:07 UTC)
orbiter warning: context deadline exceeded
Duration: 14.847058207s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
6.964028114s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:43:41 UTC)
orbiter warning: context deadline exceeded
Duration: 6.964054574s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X..X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...X.X...] duration:
6.089904539s trace:
https://jaeger.dev.storj.io/trace/569e628ae5f080ec
FAIL (2025-12-31 18:43:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 18:41:58.894515273 (627h12m58.765598319s) [upload=549605]: Starting upload
Dec 31 18:41:58.894670979 (627h12m58.765754025s) [upload=549605]: Done starting upload
Dec 31 18:41:58.894741373 (627h12m58.765824419s) [upload=549605]: Uploading segments...
Dec 31 18:41:58.894968426 (627h12m58.766051471s) [upload=549605]: Got next segment. Inline: false
Dec 31 18:41:58.895000678 (627h12m58.766083723s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:41:58.895005417 (627h12m58.766088452s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:41:59.142576097 (627h12m59.013659132s) [seg_pos={0 0}, upload=549605]: Begin upload segment...
Dec 31 18:41:59.142616474 (627h12m59.013699509s) [seg_pos={0 0}, upload=549605]: Done begin upload segment.
Dec 31 18:41:59.142646561 (627h12m59.013729596s) [noise=true , address=145.79.227.9:4367 , piece_id=CMVEX2F7TQKACZVGTUZK4KBXMFQD7XXAI74V3OODO5NL2XACFSCA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.142658804 (627h12m59.013741849s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143108100 (627h12m59.014191145s) [noise=true , address=145.79.227.36:4518 , piece_id=QFT4YLQTSEJ3VTT4EBVUR3VLZIVO4R7DVIAXNZKO6MM2STLMLTEA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143508552 (627h12m59.014591587s) [noise=true , address=77.90.31.11:4554 , piece_id=FZ3X3Z4Z5MYA2QCEA7CHIPGYCXCZSEZ7YQHZZYTKTNXDZWADVV3Q, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143897562 (627h12m59.014980598s) [noise=true , address=145.79.227.3:4151 , piece_id=GGZ24RWDCRYUTZVJJWPO2ISWEXKL3R3CP2QKA2OWN7S4IG5DTGWA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.144309247 (627h12m59.015392292s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.144716803 (627h12m59.015799838s) [noise=true , address=45.91.116.12:4227 , piece_id=EKWHVQXFTCXVUA6AV6RBYPD3L54BL25H4TI4ID3Z45JETCRII7XQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.145267782 (627h12m59.016350827s) [noise=true , address=145.79.227.39:4013 , piece_id=7RJMYLQS2VZVLEBP6OVSCTLQR6NMQJYJK2SEBOUDLNPCT64YEJXA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.146331367 (627h12m59.017414402s) [noise=true , address=145.79.227.10:4401 , piece_id=6QEL6U6S7V7VNQ7FMVUUKJJOJPCQBFEIDQIKF27CRDO64KAMMTKQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.147469464 (627h12m59.018552509s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.149355696 (627h12m59.020438731s) [noise=true , address=145.79.227.48:4301 , piece_id=FWBOVULEU46EGARXZQTENMNH7L24U6PVYMPX2FB7MQUUYAIERYNA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.150997522 (627h12m59.022080557s) [noise=true , address=77.90.31.14:4661 , piece_id=6L7HCNMJKMWNI7TQ4QJLDK4EINELCAOJ76KJCBIESISE47XXIXQQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.152544317 (627h12m59.023627352s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSQCEPU2YTT3BLCWCVJHZOPPUW6ISWB6UIFT6LTS4JIYLZ2XRGA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.153202681 (627h12m59.024285726s) [noise=true , address=45.91.116.4:4939 , piece_id=XG4LRLUYZJ3RCWFPHMUGTNQGLDX37GO3GY7QJXSIZNND4FCIOZOA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.154311182 (627h12m59.025394217s) [noise=true , address=145.79.227.13:4804 , piece_id=YPVEEGWT3UOIR25TK77OWHMKBYYWQDSOR4XICKLAVD2NWNYEH3IQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.154680856 (627h12m59.025763901s) [noise=true , address=145.79.227.16:4910 , piece_id=CHICIGXVZ4YTYDHQWO4RHQC2NBGXMXUQDIB6CVLDXAMLVUEBN6TA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:42:00.846781670 (627h13m0.717864715s) [upload=549605]: Next returned nil segment
Dec 31 18:42:00.846798912 (627h13m0.717881957s) [upload=549605]: Waiting for error group managing segments...
Dec 31 18:42:01.129758347 (627h13m1.000841392s) [noise=true , address=145.79.227.3:4151 , piece_id=GGZ24RWDCRYUTZVJJWPO2ISWEXKL3R3CP2QKA2OWN7S4IG5DTGWA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.142281002 (627h13m1.013364057s) [noise=true , address=145.79.227.48:4301 , piece_id=FWBOVULEU46EGARXZQTENMNH7L24U6PVYMPX2FB7MQUUYAIERYNA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.144383255 (627h13m1.0154663s) [noise=true , address=145.79.227.16:4910 , piece_id=CHICIGXVZ4YTYDHQWO4RHQC2NBGXMXUQDIB6CVLDXAMLVUEBN6TA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.146466080 (627h13m1.017549126s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSQCEPU2YTT3BLCWCVJHZOPPUW6ISWB6UIFT6LTS4JIYLZ2XRGA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.149361363 (627h13m1.020444418s) [noise=true , address=145.79.227.9:4367 , piece_id=CMVEX2F7TQKACZVGTUZK4KBXMFQD7XXAI74V3OODO5NL2XACFSCA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.152618416 (627h13m1.023701461s) [noise=true , address=145.79.227.10:4401 , piece_id=6QEL6U6S7V7VNQ7FMVUUKJJOJPCQBFEIDQIKF27CRDO64KAMMTKQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.157002183 (627h13m1.028085228s) [noise=true , address=145.79.227.36:4518 , piece_id=QFT4YLQTSEJ3VTT4EBVUR3VLZIVO4R7DVIAXNZKO6MM2STLMLTEA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.222096892 (627h13m1.093179927s) [noise=true , address=145.79.227.39:4013 , piece_id=7RJMYLQS2VZVLEBP6OVSCTLQR6NMQJYJK2SEBOUDLNPCT64YEJXA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.276175595 (627h13m1.14725864s) [noise=true , address=145.79.227.13:4804 , piece_id=YPVEEGWT3UOIR25TK77OWHMKBYYWQDSOR4XICKLAVD2NWNYEH3IQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.299700379 (627h13m1.170783424s) [noise=true , address=77.90.31.14:4661 , piece_id=6L7HCNMJKMWNI7TQ4QJLDK4EINELCAOJ76KJCBIESISE47XXIXQQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.451781732 (627h13m1.322864767s) [noise=true , address=45.91.116.12:4227 , piece_id=EKWHVQXFTCXVUA6AV6RBYPD3L54BL25H4TI4ID3Z45JETCRII7XQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.514670742 (627h13m1.385753797s) [noise=true , address=77.90.31.11:4554 , piece_id=FZ3X3Z4Z5MYA2QCEA7CHIPGYCXCZSEZ7YQHZZYTKTNXDZWADVV3Q, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.550595368 (627h13m1.421678403s) [noise=true , address=45.91.116.4:4939 , piece_id=XG4LRLUYZJ3RCWFPHMUGTNQGLDX37GO3GY7QJXSIZNND4FCIOZOA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.550626077 (627h13m1.421709122s) [seg_pos={0 0}, upload=549605]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:42:01.550815287 (627h13m1.421898322s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.550823031 (627h13m1.421906076s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551102934 (627h13m1.422185979s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.551110970 (627h13m1.422194015s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551281675 (627h13m1.42236472s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.551289420 (627h13m1.422372465s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551306612 (627h13m1.422389657s) [seg_pos={0 0}, upload=549605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:42:01.551377066 (627h13m1.422460111s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:42:01.551382586 (627h13m1.422465632s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:42:01.851405202 (627h13m1.722488247s) [upload=549605]: Done uploading segments...
Dec 31 18:42:01.851430740 (627h13m1.722513785s) [upload=549605]: Upload finished. err: <nil>
Duration: 6.089932131s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [....X...X.X...................X.............................] duration:
4.451332947s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:43:28 UTC)
orbiter warning: context deadline exceeded
Duration: 4.451364194s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [XX..X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...X.X..] duration:
5.84884728s trace:
https://jaeger.dev.storj.io/trace/569e628ae5f080ec
FAIL (2025-12-31 18:43:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 18:41:58.894515273 (627h12m58.765598319s) [upload=549605]: Starting upload
Dec 31 18:41:58.894670979 (627h12m58.765754025s) [upload=549605]: Done starting upload
Dec 31 18:41:58.894741373 (627h12m58.765824419s) [upload=549605]: Uploading segments...
Dec 31 18:41:58.894968426 (627h12m58.766051471s) [upload=549605]: Got next segment. Inline: false
Dec 31 18:41:58.895000678 (627h12m58.766083723s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:41:58.895005417 (627h12m58.766088452s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:41:59.142576097 (627h12m59.013659132s) [seg_pos={0 0}, upload=549605]: Begin upload segment...
Dec 31 18:41:59.142616474 (627h12m59.013699509s) [seg_pos={0 0}, upload=549605]: Done begin upload segment.
Dec 31 18:41:59.142646561 (627h12m59.013729596s) [noise=true , address=145.79.227.9:4367 , piece_id=CMVEX2F7TQKACZVGTUZK4KBXMFQD7XXAI74V3OODO5NL2XACFSCA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.142658804 (627h12m59.013741849s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143108100 (627h12m59.014191145s) [noise=true , address=145.79.227.36:4518 , piece_id=QFT4YLQTSEJ3VTT4EBVUR3VLZIVO4R7DVIAXNZKO6MM2STLMLTEA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143508552 (627h12m59.014591587s) [noise=true , address=77.90.31.11:4554 , piece_id=FZ3X3Z4Z5MYA2QCEA7CHIPGYCXCZSEZ7YQHZZYTKTNXDZWADVV3Q, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143897562 (627h12m59.014980598s) [noise=true , address=145.79.227.3:4151 , piece_id=GGZ24RWDCRYUTZVJJWPO2ISWEXKL3R3CP2QKA2OWN7S4IG5DTGWA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.144309247 (627h12m59.015392292s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.144716803 (627h12m59.015799838s) [noise=true , address=45.91.116.12:4227 , piece_id=EKWHVQXFTCXVUA6AV6RBYPD3L54BL25H4TI4ID3Z45JETCRII7XQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.145267782 (627h12m59.016350827s) [noise=true , address=145.79.227.39:4013 , piece_id=7RJMYLQS2VZVLEBP6OVSCTLQR6NMQJYJK2SEBOUDLNPCT64YEJXA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.146331367 (627h12m59.017414402s) [noise=true , address=145.79.227.10:4401 , piece_id=6QEL6U6S7V7VNQ7FMVUUKJJOJPCQBFEIDQIKF27CRDO64KAMMTKQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.147469464 (627h12m59.018552509s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.149355696 (627h12m59.020438731s) [noise=true , address=145.79.227.48:4301 , piece_id=FWBOVULEU46EGARXZQTENMNH7L24U6PVYMPX2FB7MQUUYAIERYNA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.150997522 (627h12m59.022080557s) [noise=true , address=77.90.31.14:4661 , piece_id=6L7HCNMJKMWNI7TQ4QJLDK4EINELCAOJ76KJCBIESISE47XXIXQQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.152544317 (627h12m59.023627352s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSQCEPU2YTT3BLCWCVJHZOPPUW6ISWB6UIFT6LTS4JIYLZ2XRGA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.153202681 (627h12m59.024285726s) [noise=true , address=45.91.116.4:4939 , piece_id=XG4LRLUYZJ3RCWFPHMUGTNQGLDX37GO3GY7QJXSIZNND4FCIOZOA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.154311182 (627h12m59.025394217s) [noise=true , address=145.79.227.13:4804 , piece_id=YPVEEGWT3UOIR25TK77OWHMKBYYWQDSOR4XICKLAVD2NWNYEH3IQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.154680856 (627h12m59.025763901s) [noise=true , address=145.79.227.16:4910 , piece_id=CHICIGXVZ4YTYDHQWO4RHQC2NBGXMXUQDIB6CVLDXAMLVUEBN6TA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:42:00.846781670 (627h13m0.717864715s) [upload=549605]: Next returned nil segment
Dec 31 18:42:00.846798912 (627h13m0.717881957s) [upload=549605]: Waiting for error group managing segments...
Dec 31 18:42:01.129758347 (627h13m1.000841392s) [noise=true , address=145.79.227.3:4151 , piece_id=GGZ24RWDCRYUTZVJJWPO2ISWEXKL3R3CP2QKA2OWN7S4IG5DTGWA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.142281002 (627h13m1.013364057s) [noise=true , address=145.79.227.48:4301 , piece_id=FWBOVULEU46EGARXZQTENMNH7L24U6PVYMPX2FB7MQUUYAIERYNA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.144383255 (627h13m1.0154663s) [noise=true , address=145.79.227.16:4910 , piece_id=CHICIGXVZ4YTYDHQWO4RHQC2NBGXMXUQDIB6CVLDXAMLVUEBN6TA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.146466080 (627h13m1.017549126s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSQCEPU2YTT3BLCWCVJHZOPPUW6ISWB6UIFT6LTS4JIYLZ2XRGA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.149361363 (627h13m1.020444418s) [noise=true , address=145.79.227.9:4367 , piece_id=CMVEX2F7TQKACZVGTUZK4KBXMFQD7XXAI74V3OODO5NL2XACFSCA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.152618416 (627h13m1.023701461s) [noise=true , address=145.79.227.10:4401 , piece_id=6QEL6U6S7V7VNQ7FMVUUKJJOJPCQBFEIDQIKF27CRDO64KAMMTKQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.157002183 (627h13m1.028085228s) [noise=true , address=145.79.227.36:4518 , piece_id=QFT4YLQTSEJ3VTT4EBVUR3VLZIVO4R7DVIAXNZKO6MM2STLMLTEA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.222096892 (627h13m1.093179927s) [noise=true , address=145.79.227.39:4013 , piece_id=7RJMYLQS2VZVLEBP6OVSCTLQR6NMQJYJK2SEBOUDLNPCT64YEJXA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.276175595 (627h13m1.14725864s) [noise=true , address=145.79.227.13:4804 , piece_id=YPVEEGWT3UOIR25TK77OWHMKBYYWQDSOR4XICKLAVD2NWNYEH3IQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.299700379 (627h13m1.170783424s) [noise=true , address=77.90.31.14:4661 , piece_id=6L7HCNMJKMWNI7TQ4QJLDK4EINELCAOJ76KJCBIESISE47XXIXQQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.451781732 (627h13m1.322864767s) [noise=true , address=45.91.116.12:4227 , piece_id=EKWHVQXFTCXVUA6AV6RBYPD3L54BL25H4TI4ID3Z45JETCRII7XQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.514670742 (627h13m1.385753797s) [noise=true , address=77.90.31.11:4554 , piece_id=FZ3X3Z4Z5MYA2QCEA7CHIPGYCXCZSEZ7YQHZZYTKTNXDZWADVV3Q, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.550595368 (627h13m1.421678403s) [noise=true , address=45.91.116.4:4939 , piece_id=XG4LRLUYZJ3RCWFPHMUGTNQGLDX37GO3GY7QJXSIZNND4FCIOZOA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.550626077 (627h13m1.421709122s) [seg_pos={0 0}, upload=549605]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:42:01.550815287 (627h13m1.421898322s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.550823031 (627h13m1.421906076s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551102934 (627h13m1.422185979s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.551110970 (627h13m1.422194015s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551281675 (627h13m1.42236472s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.551289420 (627h13m1.422372465s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551306612 (627h13m1.422389657s) [seg_pos={0 0}, upload=549605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:42:01.551377066 (627h13m1.422460111s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:42:01.551382586 (627h13m1.422465632s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:42:01.851405202 (627h13m1.722488247s) [upload=549605]: Done uploading segments...
Dec 31 18:42:01.851430740 (627h13m1.722513785s) [upload=549605]: Upload finished. err: <nil>
Duration: 5.84888419s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
5.981821252s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:43:04 UTC)
orbiter warning: context deadline exceeded
Duration: 5.981882959s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [XXX..X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...X.X.] duration:
6.134665371s trace:
https://jaeger.dev.storj.io/trace/569e628ae5f080ec
FAIL (2025-12-31 18:42:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 18:41:58.894515273 (627h12m58.765598319s) [upload=549605]: Starting upload
Dec 31 18:41:58.894670979 (627h12m58.765754025s) [upload=549605]: Done starting upload
Dec 31 18:41:58.894741373 (627h12m58.765824419s) [upload=549605]: Uploading segments...
Dec 31 18:41:58.894968426 (627h12m58.766051471s) [upload=549605]: Got next segment. Inline: false
Dec 31 18:41:58.895000678 (627h12m58.766083723s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:41:58.895005417 (627h12m58.766088452s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:41:59.142576097 (627h12m59.013659132s) [seg_pos={0 0}, upload=549605]: Begin upload segment...
Dec 31 18:41:59.142616474 (627h12m59.013699509s) [seg_pos={0 0}, upload=549605]: Done begin upload segment.
Dec 31 18:41:59.142646561 (627h12m59.013729596s) [noise=true , address=145.79.227.9:4367 , piece_id=CMVEX2F7TQKACZVGTUZK4KBXMFQD7XXAI74V3OODO5NL2XACFSCA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.142658804 (627h12m59.013741849s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143108100 (627h12m59.014191145s) [noise=true , address=145.79.227.36:4518 , piece_id=QFT4YLQTSEJ3VTT4EBVUR3VLZIVO4R7DVIAXNZKO6MM2STLMLTEA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143508552 (627h12m59.014591587s) [noise=true , address=77.90.31.11:4554 , piece_id=FZ3X3Z4Z5MYA2QCEA7CHIPGYCXCZSEZ7YQHZZYTKTNXDZWADVV3Q, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143897562 (627h12m59.014980598s) [noise=true , address=145.79.227.3:4151 , piece_id=GGZ24RWDCRYUTZVJJWPO2ISWEXKL3R3CP2QKA2OWN7S4IG5DTGWA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.144309247 (627h12m59.015392292s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.144716803 (627h12m59.015799838s) [noise=true , address=45.91.116.12:4227 , piece_id=EKWHVQXFTCXVUA6AV6RBYPD3L54BL25H4TI4ID3Z45JETCRII7XQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.145267782 (627h12m59.016350827s) [noise=true , address=145.79.227.39:4013 , piece_id=7RJMYLQS2VZVLEBP6OVSCTLQR6NMQJYJK2SEBOUDLNPCT64YEJXA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.146331367 (627h12m59.017414402s) [noise=true , address=145.79.227.10:4401 , piece_id=6QEL6U6S7V7VNQ7FMVUUKJJOJPCQBFEIDQIKF27CRDO64KAMMTKQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.147469464 (627h12m59.018552509s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.149355696 (627h12m59.020438731s) [noise=true , address=145.79.227.48:4301 , piece_id=FWBOVULEU46EGARXZQTENMNH7L24U6PVYMPX2FB7MQUUYAIERYNA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.150997522 (627h12m59.022080557s) [noise=true , address=77.90.31.14:4661 , piece_id=6L7HCNMJKMWNI7TQ4QJLDK4EINELCAOJ76KJCBIESISE47XXIXQQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.152544317 (627h12m59.023627352s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSQCEPU2YTT3BLCWCVJHZOPPUW6ISWB6UIFT6LTS4JIYLZ2XRGA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.153202681 (627h12m59.024285726s) [noise=true , address=45.91.116.4:4939 , piece_id=XG4LRLUYZJ3RCWFPHMUGTNQGLDX37GO3GY7QJXSIZNND4FCIOZOA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.154311182 (627h12m59.025394217s) [noise=true , address=145.79.227.13:4804 , piece_id=YPVEEGWT3UOIR25TK77OWHMKBYYWQDSOR4XICKLAVD2NWNYEH3IQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.154680856 (627h12m59.025763901s) [noise=true , address=145.79.227.16:4910 , piece_id=CHICIGXVZ4YTYDHQWO4RHQC2NBGXMXUQDIB6CVLDXAMLVUEBN6TA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:42:00.846781670 (627h13m0.717864715s) [upload=549605]: Next returned nil segment
Dec 31 18:42:00.846798912 (627h13m0.717881957s) [upload=549605]: Waiting for error group managing segments...
Dec 31 18:42:01.129758347 (627h13m1.000841392s) [noise=true , address=145.79.227.3:4151 , piece_id=GGZ24RWDCRYUTZVJJWPO2ISWEXKL3R3CP2QKA2OWN7S4IG5DTGWA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.142281002 (627h13m1.013364057s) [noise=true , address=145.79.227.48:4301 , piece_id=FWBOVULEU46EGARXZQTENMNH7L24U6PVYMPX2FB7MQUUYAIERYNA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.144383255 (627h13m1.0154663s) [noise=true , address=145.79.227.16:4910 , piece_id=CHICIGXVZ4YTYDHQWO4RHQC2NBGXMXUQDIB6CVLDXAMLVUEBN6TA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.146466080 (627h13m1.017549126s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSQCEPU2YTT3BLCWCVJHZOPPUW6ISWB6UIFT6LTS4JIYLZ2XRGA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.149361363 (627h13m1.020444418s) [noise=true , address=145.79.227.9:4367 , piece_id=CMVEX2F7TQKACZVGTUZK4KBXMFQD7XXAI74V3OODO5NL2XACFSCA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.152618416 (627h13m1.023701461s) [noise=true , address=145.79.227.10:4401 , piece_id=6QEL6U6S7V7VNQ7FMVUUKJJOJPCQBFEIDQIKF27CRDO64KAMMTKQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.157002183 (627h13m1.028085228s) [noise=true , address=145.79.227.36:4518 , piece_id=QFT4YLQTSEJ3VTT4EBVUR3VLZIVO4R7DVIAXNZKO6MM2STLMLTEA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.222096892 (627h13m1.093179927s) [noise=true , address=145.79.227.39:4013 , piece_id=7RJMYLQS2VZVLEBP6OVSCTLQR6NMQJYJK2SEBOUDLNPCT64YEJXA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.276175595 (627h13m1.14725864s) [noise=true , address=145.79.227.13:4804 , piece_id=YPVEEGWT3UOIR25TK77OWHMKBYYWQDSOR4XICKLAVD2NWNYEH3IQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.299700379 (627h13m1.170783424s) [noise=true , address=77.90.31.14:4661 , piece_id=6L7HCNMJKMWNI7TQ4QJLDK4EINELCAOJ76KJCBIESISE47XXIXQQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.451781732 (627h13m1.322864767s) [noise=true , address=45.91.116.12:4227 , piece_id=EKWHVQXFTCXVUA6AV6RBYPD3L54BL25H4TI4ID3Z45JETCRII7XQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.514670742 (627h13m1.385753797s) [noise=true , address=77.90.31.11:4554 , piece_id=FZ3X3Z4Z5MYA2QCEA7CHIPGYCXCZSEZ7YQHZZYTKTNXDZWADVV3Q, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.550595368 (627h13m1.421678403s) [noise=true , address=45.91.116.4:4939 , piece_id=XG4LRLUYZJ3RCWFPHMUGTNQGLDX37GO3GY7QJXSIZNND4FCIOZOA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.550626077 (627h13m1.421709122s) [seg_pos={0 0}, upload=549605]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:42:01.550815287 (627h13m1.421898322s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.550823031 (627h13m1.421906076s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551102934 (627h13m1.422185979s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.551110970 (627h13m1.422194015s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551281675 (627h13m1.42236472s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.551289420 (627h13m1.422372465s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551306612 (627h13m1.422389657s) [seg_pos={0 0}, upload=549605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:42:01.551377066 (627h13m1.422460111s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:42:01.551382586 (627h13m1.422465632s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:42:01.851405202 (627h13m1.722488247s) [upload=549605]: Done uploading segments...
Dec 31 18:42:01.851430740 (627h13m1.722513785s) [upload=549605]: Upload finished. err: <nil>
Duration: 6.134693976s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.....X...X.X...................X............................] duration:
3.434583449s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:42:48 UTC)
orbiter warning: context deadline exceeded
Duration: 3.434606712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
15.767153675s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:42:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 15.76718315s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
26 out of 60 is failed (threshold=3) [XXXX..X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...X.X] duration:
30.002175814s trace:
https://jaeger.dev.storj.io/trace/569e628ae5f080ec
FAIL (2025-12-31 18:42:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 18:41:58.894515273 (627h12m58.765598319s) [upload=549605]: Starting upload
Dec 31 18:41:58.894670979 (627h12m58.765754025s) [upload=549605]: Done starting upload
Dec 31 18:41:58.894741373 (627h12m58.765824419s) [upload=549605]: Uploading segments...
Dec 31 18:41:58.894968426 (627h12m58.766051471s) [upload=549605]: Got next segment. Inline: false
Dec 31 18:41:58.895000678 (627h12m58.766083723s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:41:58.895005417 (627h12m58.766088452s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:41:59.142576097 (627h12m59.013659132s) [seg_pos={0 0}, upload=549605]: Begin upload segment...
Dec 31 18:41:59.142616474 (627h12m59.013699509s) [seg_pos={0 0}, upload=549605]: Done begin upload segment.
Dec 31 18:41:59.142646561 (627h12m59.013729596s) [noise=true , address=145.79.227.9:4367 , piece_id=CMVEX2F7TQKACZVGTUZK4KBXMFQD7XXAI74V3OODO5NL2XACFSCA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.142658804 (627h12m59.013741849s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143108100 (627h12m59.014191145s) [noise=true , address=145.79.227.36:4518 , piece_id=QFT4YLQTSEJ3VTT4EBVUR3VLZIVO4R7DVIAXNZKO6MM2STLMLTEA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143508552 (627h12m59.014591587s) [noise=true , address=77.90.31.11:4554 , piece_id=FZ3X3Z4Z5MYA2QCEA7CHIPGYCXCZSEZ7YQHZZYTKTNXDZWADVV3Q, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.143897562 (627h12m59.014980598s) [noise=true , address=145.79.227.3:4151 , piece_id=GGZ24RWDCRYUTZVJJWPO2ISWEXKL3R3CP2QKA2OWN7S4IG5DTGWA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.144309247 (627h12m59.015392292s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.144716803 (627h12m59.015799838s) [noise=true , address=45.91.116.12:4227 , piece_id=EKWHVQXFTCXVUA6AV6RBYPD3L54BL25H4TI4ID3Z45JETCRII7XQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.145267782 (627h12m59.016350827s) [noise=true , address=145.79.227.39:4013 , piece_id=7RJMYLQS2VZVLEBP6OVSCTLQR6NMQJYJK2SEBOUDLNPCT64YEJXA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.146331367 (627h12m59.017414402s) [noise=true , address=145.79.227.10:4401 , piece_id=6QEL6U6S7V7VNQ7FMVUUKJJOJPCQBFEIDQIKF27CRDO64KAMMTKQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.147469464 (627h12m59.018552509s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.149355696 (627h12m59.020438731s) [noise=true , address=145.79.227.48:4301 , piece_id=FWBOVULEU46EGARXZQTENMNH7L24U6PVYMPX2FB7MQUUYAIERYNA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.150997522 (627h12m59.022080557s) [noise=true , address=77.90.31.14:4661 , piece_id=6L7HCNMJKMWNI7TQ4QJLDK4EINELCAOJ76KJCBIESISE47XXIXQQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.152544317 (627h12m59.023627352s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSQCEPU2YTT3BLCWCVJHZOPPUW6ISWB6UIFT6LTS4JIYLZ2XRGA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.153202681 (627h12m59.024285726s) [noise=true , address=45.91.116.4:4939 , piece_id=XG4LRLUYZJ3RCWFPHMUGTNQGLDX37GO3GY7QJXSIZNND4FCIOZOA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.154311182 (627h12m59.025394217s) [noise=true , address=145.79.227.13:4804 , piece_id=YPVEEGWT3UOIR25TK77OWHMKBYYWQDSOR4XICKLAVD2NWNYEH3IQ, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:41:59.154680856 (627h12m59.025763901s) [noise=true , address=145.79.227.16:4910 , piece_id=CHICIGXVZ4YTYDHQWO4RHQC2NBGXMXUQDIB6CVLDXAMLVUEBN6TA, seg_pos={0 0}, upload=549605]: Uploading piece...
Dec 31 18:42:00.846781670 (627h13m0.717864715s) [upload=549605]: Next returned nil segment
Dec 31 18:42:00.846798912 (627h13m0.717881957s) [upload=549605]: Waiting for error group managing segments...
Dec 31 18:42:01.129758347 (627h13m1.000841392s) [noise=true , address=145.79.227.3:4151 , piece_id=GGZ24RWDCRYUTZVJJWPO2ISWEXKL3R3CP2QKA2OWN7S4IG5DTGWA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.142281002 (627h13m1.013364057s) [noise=true , address=145.79.227.48:4301 , piece_id=FWBOVULEU46EGARXZQTENMNH7L24U6PVYMPX2FB7MQUUYAIERYNA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.144383255 (627h13m1.0154663s) [noise=true , address=145.79.227.16:4910 , piece_id=CHICIGXVZ4YTYDHQWO4RHQC2NBGXMXUQDIB6CVLDXAMLVUEBN6TA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.146466080 (627h13m1.017549126s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSQCEPU2YTT3BLCWCVJHZOPPUW6ISWB6UIFT6LTS4JIYLZ2XRGA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.149361363 (627h13m1.020444418s) [noise=true , address=145.79.227.9:4367 , piece_id=CMVEX2F7TQKACZVGTUZK4KBXMFQD7XXAI74V3OODO5NL2XACFSCA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.152618416 (627h13m1.023701461s) [noise=true , address=145.79.227.10:4401 , piece_id=6QEL6U6S7V7VNQ7FMVUUKJJOJPCQBFEIDQIKF27CRDO64KAMMTKQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.157002183 (627h13m1.028085228s) [noise=true , address=145.79.227.36:4518 , piece_id=QFT4YLQTSEJ3VTT4EBVUR3VLZIVO4R7DVIAXNZKO6MM2STLMLTEA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.222096892 (627h13m1.093179927s) [noise=true , address=145.79.227.39:4013 , piece_id=7RJMYLQS2VZVLEBP6OVSCTLQR6NMQJYJK2SEBOUDLNPCT64YEJXA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.276175595 (627h13m1.14725864s) [noise=true , address=145.79.227.13:4804 , piece_id=YPVEEGWT3UOIR25TK77OWHMKBYYWQDSOR4XICKLAVD2NWNYEH3IQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.299700379 (627h13m1.170783424s) [noise=true , address=77.90.31.14:4661 , piece_id=6L7HCNMJKMWNI7TQ4QJLDK4EINELCAOJ76KJCBIESISE47XXIXQQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.451781732 (627h13m1.322864767s) [noise=true , address=45.91.116.12:4227 , piece_id=EKWHVQXFTCXVUA6AV6RBYPD3L54BL25H4TI4ID3Z45JETCRII7XQ, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.514670742 (627h13m1.385753797s) [noise=true , address=77.90.31.11:4554 , piece_id=FZ3X3Z4Z5MYA2QCEA7CHIPGYCXCZSEZ7YQHZZYTKTNXDZWADVV3Q, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.550595368 (627h13m1.421678403s) [noise=true , address=45.91.116.4:4939 , piece_id=XG4LRLUYZJ3RCWFPHMUGTNQGLDX37GO3GY7QJXSIZNND4FCIOZOA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: <nil>
Dec 31 18:42:01.550626077 (627h13m1.421709122s) [seg_pos={0 0}, upload=549605]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:42:01.550815287 (627h13m1.421898322s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.550823031 (627h13m1.421906076s) [noise=true , address=45.91.116.11:4120 , piece_id=A3MXPOZFZGS5VHATFXAF5KAADLNYPPSSBUI6D7EDVDBKGSAIG74A, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551102934 (627h13m1.422185979s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.551110970 (627h13m1.422194015s) [noise=true , address=45.91.116.8:4047 , piece_id=PKSE64UIFYK25OG5P6QVSP6GSOPAQNWKJ3V23QLTPJPKHEMXOVVA, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551281675 (627h13m1.42236472s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 31 18:42:01.551289420 (627h13m1.422372465s) [noise=true , address=45.91.116.13:4155 , piece_id=V75P4EHFCPUVDLSTC7Z4LTTBEFK4QN46SEQOVPI2YQDDMY526LMA, seg_pos={0 0}, upload=549605]: Long tail cancellation due to optimal threshold
Dec 31 18:42:01.551306612 (627h13m1.422389657s) [seg_pos={0 0}, upload=549605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:42:01.551377066 (627h13m1.422460111s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:42:01.551382586 (627h13m1.422465632s) [upload=549605]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:42:01.851405202 (627h13m1.722488247s) [upload=549605]: Done uploading segments...
Dec 31 18:42:01.851430740 (627h13m1.722513785s) [upload=549605]: Upload finished. err: <nil>
Duration: 30.002198496s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
15.344113353s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:42:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 15.34413758s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
6.145787705s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:42:10 UTC)
orbiter warning: context deadline exceeded
Duration: 6.145816088s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
47.395784751s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:42:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 47.395810422s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [......X...X.X...................X...........................] duration:
3.729971714s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:42:02 UTC)
orbiter warning: context deadline exceeded
Duration: 3.729996881s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [.XXXX..X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...X.] duration:
8.809708839s trace:
https://jaeger.dev.storj.io/trace/371afc6375d18b71
FAIL (2025-12-31 18:41:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 18:41:10.326279393 (627h12m10.197362438s) [upload=549590]: Starting upload
Dec 31 18:41:10.326408179 (627h12m10.197491224s) [upload=549590]: Done starting upload
Dec 31 18:41:10.326499352 (627h12m10.197582397s) [upload=549590]: Uploading segments...
Dec 31 18:41:10.326890988 (627h12m10.197974033s) [upload=549590]: Got next segment. Inline: false
Dec 31 18:41:10.326975018 (627h12m10.198058063s) [upload=549590]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:41:10.326981149 (627h12m10.198064204s) [upload=549590]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:41:10.582560749 (627h12m10.453643824s) [seg_pos={0 0}, upload=549590]: Begin upload segment...
Dec 31 18:41:10.582642955 (627h12m10.453726s) [seg_pos={0 0}, upload=549590]: Done begin upload segment.
Dec 31 18:41:10.582683121 (627h12m10.453766157s) [noise=true , address=77.90.31.13:4629 , piece_id=YGE2UF5SUDMOJ4CEXG2OTIII7OF56ANQOA4DWOHRTQ4N35XUGDYA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.582689384 (627h12m10.453772429s) [noise=true , address=145.79.227.21:4475 , piece_id=XDRETDQLS6UHENTQ2S2WGGAY4Q6IHML6NFEIITJLNZVF7DNBRJPA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.583547868 (627h12m10.454630913s) [noise=true , address=145.79.227.7:4293 , piece_id=XOCSD5ZX4NSLZ2WWHZK5QM56GXRAZQ4GBLAVQ7XDCJYLD3L6XLAA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.583834434 (627h12m10.454917479s) [noise=true , address=145.79.227.48:4302 , piece_id=JQLSRUWJ33HK6WMLTISY6EXLHVXXBEZXLDJLSPGSLDKNXP44F2HA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.583982346 (627h12m10.455065381s) [noise=true , address=145.79.227.39:4013 , piece_id=SD6BLDJHBYPNIOHSPDUWGMAPMGR22V5L3I7YA5XHVBLF5GXAII3Q, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.584283268 (627h12m10.455366314s) [noise=true , address=45.91.116.8:4045 , piece_id=PE4MHFQX5ABO2N3HYU3CH3XWQ4MXHR6BSPQNUXG2FMZWL6YUY6XQ, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.584374643 (627h12m10.455457678s) [noise=true , address=145.79.227.43:4158 , piece_id=5OH6KHDNOQGAUK3CIL3SZ5VFIVOYKNKJGXBGVYMPW3KEKMRZBTAA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.584548925 (627h12m10.45563196s) [noise=true , address=145.79.227.11:4729 , piece_id=AYR6BCCVWL4VWRWCDAJM6HAQNEINTB3ILH7QJKIDAJPP7HY7UE4Q, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.584934148 (627h12m10.456017193s) [noise=true , address=145.79.227.52:4445 , piece_id=6VOMSBPM2FJVK5XRPTNZZRI4ONUZ2AGGYJC3KB2TGAOHGODBGD2Q, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.585551835 (627h12m10.45663488s) [noise=true , address=145.79.227.45:4230 , piece_id=ECHHPQVCEW7S22HG6EYLSD6NEB5WI3U6RRBFQIJZZIB27UXMFNTQ, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.585906971 (627h12m10.456990006s) [noise=true , address=145.79.227.34:4906 , piece_id=OHIS22S6W4LP6S5BZRU3DHTFV3ZLMIPPQPENBS4HXDZXI6JA2KOQ, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.586246107 (627h12m10.457329142s) [noise=true , address=45.91.116.14:4192 , piece_id=TNSWLFG3AFMFRWHFCQT73B27DEQXNYU4BBQOI56BQWVYWC3SNPUQ, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.586553853 (627h12m10.457636888s) [noise=true , address=145.79.227.42:4123 , piece_id=AMUS57G4ANO4YEZUFY7UWLEM2QOP7H4XLZJ5PY6ZVTT33GM4HVMA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.586933274 (627h12m10.458016309s) [noise=true , address=145.79.227.6:4258 , piece_id=2PKQBZK3AF7OTPSWAU5HADZBTFEBIUNQFJ5YQCIDAYEMR5W5SDFA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.587278151 (627h12m10.458361196s) [noise=true , address=145.79.227.50:4375 , piece_id=DEUPBYRMOMRQH6RWTBIMHTH2RLVGXEFT73P2GOROJZCKHTI4DUFQ, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.587494965 (627h12m10.45857801s) [noise=true , address=77.90.31.19:4841 , piece_id=KPTMWHP3F5G546H5ZBEGZNCWBMY4W7JIVVJUVO2CR6LWBRU3AONA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:12.285395294 (627h12m12.156478339s) [upload=549590]: Next returned nil segment
Dec 31 18:41:12.285423628 (627h12m12.156506683s) [upload=549590]: Waiting for error group managing segments...
Dec 31 18:41:12.556805577 (627h12m12.427888612s) [noise=true , address=145.79.227.21:4475 , piece_id=XDRETDQLS6UHENTQ2S2WGGAY4Q6IHML6NFEIITJLNZVF7DNBRJPA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.574784542 (627h12m12.445867577s) [noise=true , address=145.79.227.42:4123 , piece_id=AMUS57G4ANO4YEZUFY7UWLEM2QOP7H4XLZJ5PY6ZVTT33GM4HVMA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.578140482 (627h12m12.449223517s) [noise=true , address=145.79.227.45:4230 , piece_id=ECHHPQVCEW7S22HG6EYLSD6NEB5WI3U6RRBFQIJZZIB27UXMFNTQ, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.578549741 (627h12m12.449632776s) [noise=true , address=145.79.227.11:4729 , piece_id=AYR6BCCVWL4VWRWCDAJM6HAQNEINTB3ILH7QJKIDAJPP7HY7UE4Q, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.579223384 (627h12m12.450306429s) [noise=true , address=145.79.227.52:4445 , piece_id=6VOMSBPM2FJVK5XRPTNZZRI4ONUZ2AGGYJC3KB2TGAOHGODBGD2Q, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.585138868 (627h12m12.456221913s) [noise=true , address=145.79.227.6:4258 , piece_id=2PKQBZK3AF7OTPSWAU5HADZBTFEBIUNQFJ5YQCIDAYEMR5W5SDFA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.591722214 (627h12m12.462805259s) [noise=true , address=145.79.227.7:4293 , piece_id=XOCSD5ZX4NSLZ2WWHZK5QM56GXRAZQ4GBLAVQ7XDCJYLD3L6XLAA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.593435055 (627h12m12.46451809s) [noise=true , address=145.79.227.48:4302 , piece_id=JQLSRUWJ33HK6WMLTISY6EXLHVXXBEZXLDJLSPGSLDKNXP44F2HA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.661546428 (627h12m12.532629463s) [noise=true , address=145.79.227.50:4375 , piece_id=DEUPBYRMOMRQH6RWTBIMHTH2RLVGXEFT73P2GOROJZCKHTI4DUFQ, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.664488200 (627h12m12.535571235s) [noise=true , address=145.79.227.39:4013 , piece_id=SD6BLDJHBYPNIOHSPDUWGMAPMGR22V5L3I7YA5XHVBLF5GXAII3Q, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.867718519 (627h12m12.738801564s) [noise=true , address=77.90.31.19:4841 , piece_id=KPTMWHP3F5G546H5ZBEGZNCWBMY4W7JIVVJUVO2CR6LWBRU3AONA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.977609356 (627h12m12.848692391s) [noise=true , address=45.91.116.14:4192 , piece_id=TNSWLFG3AFMFRWHFCQT73B27DEQXNYU4BBQOI56BQWVYWC3SNPUQ, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:13.872899734 (627h12m13.743982769s) [noise=true , address=45.91.116.8:4045 , piece_id=PE4MHFQX5ABO2N3HYU3CH3XWQ4MXHR6BSPQNUXG2FMZWL6YUY6XQ, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:13.872918941 (627h12m13.744001976s) [seg_pos={0 0}, upload=549590]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:41:13.873426887 (627h12m13.744509933s) [noise=true , address=145.79.227.34:4906 , piece_id=OHIS22S6W4LP6S5BZRU3DHTFV3ZLMIPPQPENBS4HXDZXI6JA2KOQ, seg_pos={0 0}, upload=549590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 31 18:41:13.873441897 (627h12m13.744524962s) [noise=true , address=145.79.227.34:4906 , piece_id=OHIS22S6W4LP6S5BZRU3DHTFV3ZLMIPPQPENBS4HXDZXI6JA2KOQ, seg_pos={0 0}, upload=549590]: Long tail cancellation due to optimal threshold
Dec 31 18:41:13.873593996 (627h12m13.744677031s) [noise=true , address=145.79.227.43:4158 , piece_id=5OH6KHDNOQGAUK3CIL3SZ5VFIVOYKNKJGXBGVYMPW3KEKMRZBTAA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 31 18:41:13.873602482 (627h12m13.744685527s) [noise=true , address=145.79.227.43:4158 , piece_id=5OH6KHDNOQGAUK3CIL3SZ5VFIVOYKNKJGXBGVYMPW3KEKMRZBTAA, seg_pos={0 0}, upload=549590]: Long tail cancellation due to optimal threshold
Dec 31 18:41:13.873645453 (627h12m13.744728488s) [noise=true , address=77.90.31.13:4629 , piece_id=YGE2UF5SUDMOJ4CEXG2OTIII7OF56ANQOA4DWOHRTQ4N35XUGDYA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF
Dec 31 18:41:13.873665973 (627h12m13.744749008s) [noise=true , address=77.90.31.13:4629 , piece_id=YGE2UF5SUDMOJ4CEXG2OTIII7OF56ANQOA4DWOHRTQ4N35XUGDYA, seg_pos={0 0}, upload=549590]: Long tail cancellation due to optimal threshold
Dec 31 18:41:13.873685059 (627h12m13.744768094s) [seg_pos={0 0}, upload=549590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:41:13.873777575 (627h12m13.74486061s) [upload=549590]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:41:13.873784528 (627h12m13.744867563s) [upload=549590]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:41:14.170929233 (627h12m14.042012268s) [upload=549590]: Done uploading segments...
Dec 31 18:41:14.170948200 (627h12m14.042031245s) [upload=549590]: Upload finished. err: <nil>
Duration: 8.809723907s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [..XXXX..X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...X] duration:
30.003426518s trace:
https://jaeger.dev.storj.io/trace/371afc6375d18b71
FAIL (2025-12-31 18:41:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 18:41:10.326279393 (627h12m10.197362438s) [upload=549590]: Starting upload
Dec 31 18:41:10.326408179 (627h12m10.197491224s) [upload=549590]: Done starting upload
Dec 31 18:41:10.326499352 (627h12m10.197582397s) [upload=549590]: Uploading segments...
Dec 31 18:41:10.326890988 (627h12m10.197974033s) [upload=549590]: Got next segment. Inline: false
Dec 31 18:41:10.326975018 (627h12m10.198058063s) [upload=549590]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:41:10.326981149 (627h12m10.198064204s) [upload=549590]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:41:10.582560749 (627h12m10.453643824s) [seg_pos={0 0}, upload=549590]: Begin upload segment...
Dec 31 18:41:10.582642955 (627h12m10.453726s) [seg_pos={0 0}, upload=549590]: Done begin upload segment.
Dec 31 18:41:10.582683121 (627h12m10.453766157s) [noise=true , address=77.90.31.13:4629 , piece_id=YGE2UF5SUDMOJ4CEXG2OTIII7OF56ANQOA4DWOHRTQ4N35XUGDYA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.582689384 (627h12m10.453772429s) [noise=true , address=145.79.227.21:4475 , piece_id=XDRETDQLS6UHENTQ2S2WGGAY4Q6IHML6NFEIITJLNZVF7DNBRJPA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.583547868 (627h12m10.454630913s) [noise=true , address=145.79.227.7:4293 , piece_id=XOCSD5ZX4NSLZ2WWHZK5QM56GXRAZQ4GBLAVQ7XDCJYLD3L6XLAA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.583834434 (627h12m10.454917479s) [noise=true , address=145.79.227.48:4302 , piece_id=JQLSRUWJ33HK6WMLTISY6EXLHVXXBEZXLDJLSPGSLDKNXP44F2HA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.583982346 (627h12m10.455065381s) [noise=true , address=145.79.227.39:4013 , piece_id=SD6BLDJHBYPNIOHSPDUWGMAPMGR22V5L3I7YA5XHVBLF5GXAII3Q, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.584283268 (627h12m10.455366314s) [noise=true , address=45.91.116.8:4045 , piece_id=PE4MHFQX5ABO2N3HYU3CH3XWQ4MXHR6BSPQNUXG2FMZWL6YUY6XQ, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.584374643 (627h12m10.455457678s) [noise=true , address=145.79.227.43:4158 , piece_id=5OH6KHDNOQGAUK3CIL3SZ5VFIVOYKNKJGXBGVYMPW3KEKMRZBTAA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.584548925 (627h12m10.45563196s) [noise=true , address=145.79.227.11:4729 , piece_id=AYR6BCCVWL4VWRWCDAJM6HAQNEINTB3ILH7QJKIDAJPP7HY7UE4Q, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.584934148 (627h12m10.456017193s) [noise=true , address=145.79.227.52:4445 , piece_id=6VOMSBPM2FJVK5XRPTNZZRI4ONUZ2AGGYJC3KB2TGAOHGODBGD2Q, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.585551835 (627h12m10.45663488s) [noise=true , address=145.79.227.45:4230 , piece_id=ECHHPQVCEW7S22HG6EYLSD6NEB5WI3U6RRBFQIJZZIB27UXMFNTQ, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.585906971 (627h12m10.456990006s) [noise=true , address=145.79.227.34:4906 , piece_id=OHIS22S6W4LP6S5BZRU3DHTFV3ZLMIPPQPENBS4HXDZXI6JA2KOQ, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.586246107 (627h12m10.457329142s) [noise=true , address=45.91.116.14:4192 , piece_id=TNSWLFG3AFMFRWHFCQT73B27DEQXNYU4BBQOI56BQWVYWC3SNPUQ, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.586553853 (627h12m10.457636888s) [noise=true , address=145.79.227.42:4123 , piece_id=AMUS57G4ANO4YEZUFY7UWLEM2QOP7H4XLZJ5PY6ZVTT33GM4HVMA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.586933274 (627h12m10.458016309s) [noise=true , address=145.79.227.6:4258 , piece_id=2PKQBZK3AF7OTPSWAU5HADZBTFEBIUNQFJ5YQCIDAYEMR5W5SDFA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.587278151 (627h12m10.458361196s) [noise=true , address=145.79.227.50:4375 , piece_id=DEUPBYRMOMRQH6RWTBIMHTH2RLVGXEFT73P2GOROJZCKHTI4DUFQ, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:10.587494965 (627h12m10.45857801s) [noise=true , address=77.90.31.19:4841 , piece_id=KPTMWHP3F5G546H5ZBEGZNCWBMY4W7JIVVJUVO2CR6LWBRU3AONA, seg_pos={0 0}, upload=549590]: Uploading piece...
Dec 31 18:41:12.285395294 (627h12m12.156478339s) [upload=549590]: Next returned nil segment
Dec 31 18:41:12.285423628 (627h12m12.156506683s) [upload=549590]: Waiting for error group managing segments...
Dec 31 18:41:12.556805577 (627h12m12.427888612s) [noise=true , address=145.79.227.21:4475 , piece_id=XDRETDQLS6UHENTQ2S2WGGAY4Q6IHML6NFEIITJLNZVF7DNBRJPA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.574784542 (627h12m12.445867577s) [noise=true , address=145.79.227.42:4123 , piece_id=AMUS57G4ANO4YEZUFY7UWLEM2QOP7H4XLZJ5PY6ZVTT33GM4HVMA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.578140482 (627h12m12.449223517s) [noise=true , address=145.79.227.45:4230 , piece_id=ECHHPQVCEW7S22HG6EYLSD6NEB5WI3U6RRBFQIJZZIB27UXMFNTQ, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.578549741 (627h12m12.449632776s) [noise=true , address=145.79.227.11:4729 , piece_id=AYR6BCCVWL4VWRWCDAJM6HAQNEINTB3ILH7QJKIDAJPP7HY7UE4Q, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.579223384 (627h12m12.450306429s) [noise=true , address=145.79.227.52:4445 , piece_id=6VOMSBPM2FJVK5XRPTNZZRI4ONUZ2AGGYJC3KB2TGAOHGODBGD2Q, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.585138868 (627h12m12.456221913s) [noise=true , address=145.79.227.6:4258 , piece_id=2PKQBZK3AF7OTPSWAU5HADZBTFEBIUNQFJ5YQCIDAYEMR5W5SDFA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.591722214 (627h12m12.462805259s) [noise=true , address=145.79.227.7:4293 , piece_id=XOCSD5ZX4NSLZ2WWHZK5QM56GXRAZQ4GBLAVQ7XDCJYLD3L6XLAA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.593435055 (627h12m12.46451809s) [noise=true , address=145.79.227.48:4302 , piece_id=JQLSRUWJ33HK6WMLTISY6EXLHVXXBEZXLDJLSPGSLDKNXP44F2HA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.661546428 (627h12m12.532629463s) [noise=true , address=145.79.227.50:4375 , piece_id=DEUPBYRMOMRQH6RWTBIMHTH2RLVGXEFT73P2GOROJZCKHTI4DUFQ, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.664488200 (627h12m12.535571235s) [noise=true , address=145.79.227.39:4013 , piece_id=SD6BLDJHBYPNIOHSPDUWGMAPMGR22V5L3I7YA5XHVBLF5GXAII3Q, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.867718519 (627h12m12.738801564s) [noise=true , address=77.90.31.19:4841 , piece_id=KPTMWHP3F5G546H5ZBEGZNCWBMY4W7JIVVJUVO2CR6LWBRU3AONA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:12.977609356 (627h12m12.848692391s) [noise=true , address=45.91.116.14:4192 , piece_id=TNSWLFG3AFMFRWHFCQT73B27DEQXNYU4BBQOI56BQWVYWC3SNPUQ, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:13.872899734 (627h12m13.743982769s) [noise=true , address=45.91.116.8:4045 , piece_id=PE4MHFQX5ABO2N3HYU3CH3XWQ4MXHR6BSPQNUXG2FMZWL6YUY6XQ, seg_pos={0 0}, upload=549590]: Done uploading piece. err: <nil>
Dec 31 18:41:13.872918941 (627h12m13.744001976s) [seg_pos={0 0}, upload=549590]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:41:13.873426887 (627h12m13.744509933s) [noise=true , address=145.79.227.34:4906 , piece_id=OHIS22S6W4LP6S5BZRU3DHTFV3ZLMIPPQPENBS4HXDZXI6JA2KOQ, seg_pos={0 0}, upload=549590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 31 18:41:13.873441897 (627h12m13.744524962s) [noise=true , address=145.79.227.34:4906 , piece_id=OHIS22S6W4LP6S5BZRU3DHTFV3ZLMIPPQPENBS4HXDZXI6JA2KOQ, seg_pos={0 0}, upload=549590]: Long tail cancellation due to optimal threshold
Dec 31 18:41:13.873593996 (627h12m13.744677031s) [noise=true , address=145.79.227.43:4158 , piece_id=5OH6KHDNOQGAUK3CIL3SZ5VFIVOYKNKJGXBGVYMPW3KEKMRZBTAA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 31 18:41:13.873602482 (627h12m13.744685527s) [noise=true , address=145.79.227.43:4158 , piece_id=5OH6KHDNOQGAUK3CIL3SZ5VFIVOYKNKJGXBGVYMPW3KEKMRZBTAA, seg_pos={0 0}, upload=549590]: Long tail cancellation due to optimal threshold
Dec 31 18:41:13.873645453 (627h12m13.744728488s) [noise=true , address=77.90.31.13:4629 , piece_id=YGE2UF5SUDMOJ4CEXG2OTIII7OF56ANQOA4DWOHRTQ4N35XUGDYA, seg_pos={0 0}, upload=549590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF
Dec 31 18:41:13.873665973 (627h12m13.744749008s) [noise=true , address=77.90.31.13:4629 , piece_id=YGE2UF5SUDMOJ4CEXG2OTIII7OF56ANQOA4DWOHRTQ4N35XUGDYA, seg_pos={0 0}, upload=549590]: Long tail cancellation due to optimal threshold
Dec 31 18:41:13.873685059 (627h12m13.744768094s) [seg_pos={0 0}, upload=549590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:41:13.873777575 (627h12m13.74486061s) [upload=549590]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:41:13.873784528 (627h12m13.744867563s) [upload=549590]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:41:14.170929233 (627h12m14.042012268s) [upload=549590]: Done uploading segments...
Dec 31 18:41:14.170948200 (627h12m14.042031245s) [upload=549590]: Upload finished. err: <nil>
Duration: 30.00344908s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.......X...X.X...................X..........................] duration:
3.409621887s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:41:29 UTC)
orbiter warning: context deadline exceeded
Duration: 3.409873145s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [........X...X.X...................X.........................] duration:
3.358852294s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:41:07 UTC)
orbiter warning: context deadline exceeded
Duration: 3.358885086s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
6.038162757s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:40:58 UTC)
orbiter warning: context deadline exceeded
Duration: 6.038187854s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
39.015039157s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:40:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 39.015069565s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [X..XXXX..X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X...] duration:
6.166445321s trace:
https://jaeger.dev.storj.io/trace/543106d0f5f2249a
FAIL (2025-12-31 18:40:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 18:37:19.295757256 (627h8m19.166840291s) [upload=549537]: Starting upload
Dec 31 18:37:19.295850393 (627h8m19.166933438s) [upload=549537]: Done starting upload
Dec 31 18:37:19.295931367 (627h8m19.167014422s) [upload=549537]: Uploading segments...
Dec 31 18:37:19.296106891 (627h8m19.167189937s) [upload=549537]: Got next segment. Inline: false
Dec 31 18:37:19.296173138 (627h8m19.167256183s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:37:19.296179520 (627h8m19.167262565s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:37:19.546762337 (627h8m19.417845372s) [seg_pos={0 0}, upload=549537]: Begin upload segment...
Dec 31 18:37:19.546826458 (627h8m19.417909494s) [seg_pos={0 0}, upload=549537]: Done begin upload segment.
Dec 31 18:37:19.546990982 (627h8m19.418074017s) [noise=true , address=145.79.227.32:4836 , piece_id=6BG4P7IQ7DP7LK45ULWU7LT3RAHXP47F3VG6AB2VVQATIECJBUMQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.547127642 (627h8m19.418210697s) [noise=true , address=45.91.116.11:4120 , piece_id=VJK6C2CN7QYB62LMEPYCMYJVAQ72SVOSTP3B7NJX3XBX4OLI3XYQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.547652582 (627h8m19.418735627s) [noise=true , address=45.91.116.10:4083 , piece_id=AYWW2GAOUL64XSPDBR7N2ALI5AVAUXBZBEMUC7FTGMBQGPGUMGYA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.548335392 (627h8m19.419418427s) [noise=true , address=77.90.31.21:4878 , piece_id=PNGE43HPGT44V3HESSNNTXUS2WF3WDVSI23S4QF65NRT2YSNQIOA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.548843209 (627h8m19.419926254s) [noise=true , address=145.79.227.13:4802 , piece_id=QTFTLMTDUE2MZNH2Q546WZN53TCMSU6SDGHAYRGXC62RYUQ4JTIQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.549465824 (627h8m19.42054886s) [noise=true , address=145.79.227.52:4445 , piece_id=B77YXBCHOAQQNXJVWNSRDADDSBVWTM3QLUD5UA74L3BQ5SVG65MQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.550758155 (627h8m19.42184119s) [noise=true , address=145.79.227.26:4654 , piece_id=JZMRX6LFDNPJW7R4KVBFHGONHU3NFEEZ5GSM7NRJU5LHDSR2QG2Q, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.552324317 (627h8m19.423407352s) [noise=true , address=77.90.31.11:4553 , piece_id=ZS7BNAXMXUWHOAOJTPBTHAHF6UK6CEMK4DLLJ5N6LYMRE7G7CXSA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.553754501 (627h8m19.424837546s) [noise=true , address=145.79.227.2:4115 , piece_id=IHMXPIJARQNI2H3IFXRNEVO33ZIU3FEQRH6NGJRXVV6LS2BUKN3Q, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.554549765 (627h8m19.42563281s) [noise=true , address=145.79.227.16:4910 , piece_id=QDARBVNNNM6CTHWOSI2BWT4LVMT5XDGHBILWYNGWNHUASN553AUA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.555532045 (627h8m19.42661508s) [noise=true , address=145.79.227.10:4401 , piece_id=ZQVJJBJTT6RMZMKLLDHF3PH2S2VJN2X4Z263OEPUHE5SFH2EM4FQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.555982804 (627h8m19.427065839s) [noise=true , address=145.79.227.9:4367 , piece_id=3GZSZLVA3BQMIUPSYD56GF7G3UDIFEBJUYEGTXHPTS3WBQYEY5GQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.557313197 (627h8m19.428396232s) [noise=true , address=45.91.116.4:4939 , piece_id=VNDC3YZO4SLOAHO5MKGQJFEDSR4TDHV2Z6YABUAPZE3FBOEJZWSQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.557432204 (627h8m19.428515239s) [noise=true , address=45.91.116.5:4299 , piece_id=OOTXWS7I2QABUC3MLTNZNBNQNPZKB4VZOSVL5IQI5CLSDACVVAGA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.558589517 (627h8m19.429672552s) [noise=true , address=145.79.227.11:4730 , piece_id=WISR335RWCTCOYFIHWRSKDCDTGGCQRCR5OBYJHV5ZAAXU2ZTIJIA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.559259333 (627h8m19.430342368s) [noise=true , address=145.79.227.33:4869 , piece_id=BEZARUUS5WARJNMFI2K3FJC2BJ374KD6Z563R2SBMAWIG23VR5NA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:21.246090342 (627h8m21.117173387s) [upload=549537]: Next returned nil segment
Dec 31 18:37:21.246110060 (627h8m21.117193105s) [upload=549537]: Waiting for error group managing segments...
Dec 31 18:37:21.526859025 (627h8m21.397942071s) [noise=true , address=145.79.227.10:4401 , piece_id=ZQVJJBJTT6RMZMKLLDHF3PH2S2VJN2X4Z263OEPUHE5SFH2EM4FQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.536490225 (627h8m21.40757326s) [noise=true , address=145.79.227.52:4445 , piece_id=B77YXBCHOAQQNXJVWNSRDADDSBVWTM3QLUD5UA74L3BQ5SVG65MQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.537390770 (627h8m21.408473805s) [noise=true , address=145.79.227.2:4115 , piece_id=IHMXPIJARQNI2H3IFXRNEVO33ZIU3FEQRH6NGJRXVV6LS2BUKN3Q, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.547254842 (627h8m21.418337887s) [noise=true , address=145.79.227.13:4802 , piece_id=QTFTLMTDUE2MZNH2Q546WZN53TCMSU6SDGHAYRGXC62RYUQ4JTIQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.561016606 (627h8m21.432099651s) [noise=true , address=145.79.227.9:4367 , piece_id=3GZSZLVA3BQMIUPSYD56GF7G3UDIFEBJUYEGTXHPTS3WBQYEY5GQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.567974195 (627h8m21.43905724s) [noise=true , address=145.79.227.26:4654 , piece_id=JZMRX6LFDNPJW7R4KVBFHGONHU3NFEEZ5GSM7NRJU5LHDSR2QG2Q, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.571333481 (627h8m21.442416516s) [noise=true , address=145.79.227.16:4910 , piece_id=QDARBVNNNM6CTHWOSI2BWT4LVMT5XDGHBILWYNGWNHUASN553AUA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.701924975 (627h8m21.57300802s) [noise=true , address=145.79.227.11:4730 , piece_id=WISR335RWCTCOYFIHWRSKDCDTGGCQRCR5OBYJHV5ZAAXU2ZTIJIA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.819851494 (627h8m21.690934529s) [noise=true , address=77.90.31.11:4553 , piece_id=ZS7BNAXMXUWHOAOJTPBTHAHF6UK6CEMK4DLLJ5N6LYMRE7G7CXSA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.840948916 (627h8m21.712031951s) [noise=true , address=77.90.31.21:4878 , piece_id=PNGE43HPGT44V3HESSNNTXUS2WF3WDVSI23S4QF65NRT2YSNQIOA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.878060091 (627h8m21.749143136s) [noise=true , address=45.91.116.4:4939 , piece_id=VNDC3YZO4SLOAHO5MKGQJFEDSR4TDHV2Z6YABUAPZE3FBOEJZWSQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.974439350 (627h8m21.845522395s) [noise=true , address=45.91.116.5:4299 , piece_id=OOTXWS7I2QABUC3MLTNZNBNQNPZKB4VZOSVL5IQI5CLSDACVVAGA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.999636789 (627h8m21.870719824s) [noise=true , address=45.91.116.10:4083 , piece_id=AYWW2GAOUL64XSPDBR7N2ALI5AVAUXBZBEMUC7FTGMBQGPGUMGYA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.999658621 (627h8m21.870741656s) [seg_pos={0 0}, upload=549537]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:37:21.999841809 (627h8m21.870924854s) [noise=true , address=45.91.116.11:4120 , piece_id=VJK6C2CN7QYB62LMEPYCMYJVAQ72SVOSTP3B7NJX3XBX4OLI3XYQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:21.999850315 (627h8m21.87093338s) [noise=true , address=45.91.116.11:4120 , piece_id=VJK6C2CN7QYB62LMEPYCMYJVAQ72SVOSTP3B7NJX3XBX4OLI3XYQ, seg_pos={0 0}, upload=549537]: Long tail cancellation due to optimal threshold
Dec 31 18:37:22.000200372 (627h8m21.871283427s) [noise=true , address=145.79.227.33:4869 , piece_id=BEZARUUS5WARJNMFI2K3FJC2BJ374KD6Z563R2SBMAWIG23VR5NA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:22.000208077 (627h8m21.871291122s) [noise=true , address=145.79.227.33:4869 , piece_id=BEZARUUS5WARJNMFI2K3FJC2BJ374KD6Z563R2SBMAWIG23VR5NA, seg_pos={0 0}, upload=549537]: Long tail cancellation due to optimal threshold
Dec 31 18:37:22.000375746 (627h8m21.871458802s) [noise=true , address=145.79.227.32:4836 , piece_id=6BG4P7IQ7DP7LK45ULWU7LT3RAHXP47F3VG6AB2VVQATIECJBUMQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:22.000383060 (627h8m21.871466105s) [noise=true , address=145.79.227.32:4836 , piece_id=6BG4P7IQ7DP7LK45ULWU7LT3RAHXP47F3VG6AB2VVQATIECJBUMQ, seg_pos={0 0}, upload=549537]: Long tail cancellation due to optimal threshold
Dec 31 18:37:22.000399561 (627h8m21.871482596s) [seg_pos={0 0}, upload=549537]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:37:22.000474874 (627h8m21.871557909s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:22.000479624 (627h8m21.871562659s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:22.308546746 (627h8m22.179629791s) [upload=549537]: Done uploading segments...
Dec 31 18:37:22.308571582 (627h8m22.179654627s) [upload=549537]: Upload finished. err: <nil>
Duration: 6.166472434s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.........X...X.X...................X........................] duration:
3.433491735s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:40:26 UTC)
orbiter warning: context deadline exceeded
Duration: 3.433513917s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
5.938846123s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:40:24 UTC)
orbiter warning: context deadline exceeded
Duration: 5.93887067s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
6.132214461s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:39:54 UTC)
orbiter warning: context deadline exceeded
Duration: 6.132263235s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
17.746329796s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:39:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 17.746355565s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [.X..XXXX..X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X..] duration:
5.844829063s trace:
https://jaeger.dev.storj.io/trace/543106d0f5f2249a
FAIL (2025-12-31 18:39:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 18:37:19.295757256 (627h8m19.166840291s) [upload=549537]: Starting upload
Dec 31 18:37:19.295850393 (627h8m19.166933438s) [upload=549537]: Done starting upload
Dec 31 18:37:19.295931367 (627h8m19.167014422s) [upload=549537]: Uploading segments...
Dec 31 18:37:19.296106891 (627h8m19.167189937s) [upload=549537]: Got next segment. Inline: false
Dec 31 18:37:19.296173138 (627h8m19.167256183s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:37:19.296179520 (627h8m19.167262565s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:37:19.546762337 (627h8m19.417845372s) [seg_pos={0 0}, upload=549537]: Begin upload segment...
Dec 31 18:37:19.546826458 (627h8m19.417909494s) [seg_pos={0 0}, upload=549537]: Done begin upload segment.
Dec 31 18:37:19.546990982 (627h8m19.418074017s) [noise=true , address=145.79.227.32:4836 , piece_id=6BG4P7IQ7DP7LK45ULWU7LT3RAHXP47F3VG6AB2VVQATIECJBUMQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.547127642 (627h8m19.418210697s) [noise=true , address=45.91.116.11:4120 , piece_id=VJK6C2CN7QYB62LMEPYCMYJVAQ72SVOSTP3B7NJX3XBX4OLI3XYQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.547652582 (627h8m19.418735627s) [noise=true , address=45.91.116.10:4083 , piece_id=AYWW2GAOUL64XSPDBR7N2ALI5AVAUXBZBEMUC7FTGMBQGPGUMGYA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.548335392 (627h8m19.419418427s) [noise=true , address=77.90.31.21:4878 , piece_id=PNGE43HPGT44V3HESSNNTXUS2WF3WDVSI23S4QF65NRT2YSNQIOA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.548843209 (627h8m19.419926254s) [noise=true , address=145.79.227.13:4802 , piece_id=QTFTLMTDUE2MZNH2Q546WZN53TCMSU6SDGHAYRGXC62RYUQ4JTIQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.549465824 (627h8m19.42054886s) [noise=true , address=145.79.227.52:4445 , piece_id=B77YXBCHOAQQNXJVWNSRDADDSBVWTM3QLUD5UA74L3BQ5SVG65MQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.550758155 (627h8m19.42184119s) [noise=true , address=145.79.227.26:4654 , piece_id=JZMRX6LFDNPJW7R4KVBFHGONHU3NFEEZ5GSM7NRJU5LHDSR2QG2Q, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.552324317 (627h8m19.423407352s) [noise=true , address=77.90.31.11:4553 , piece_id=ZS7BNAXMXUWHOAOJTPBTHAHF6UK6CEMK4DLLJ5N6LYMRE7G7CXSA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.553754501 (627h8m19.424837546s) [noise=true , address=145.79.227.2:4115 , piece_id=IHMXPIJARQNI2H3IFXRNEVO33ZIU3FEQRH6NGJRXVV6LS2BUKN3Q, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.554549765 (627h8m19.42563281s) [noise=true , address=145.79.227.16:4910 , piece_id=QDARBVNNNM6CTHWOSI2BWT4LVMT5XDGHBILWYNGWNHUASN553AUA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.555532045 (627h8m19.42661508s) [noise=true , address=145.79.227.10:4401 , piece_id=ZQVJJBJTT6RMZMKLLDHF3PH2S2VJN2X4Z263OEPUHE5SFH2EM4FQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.555982804 (627h8m19.427065839s) [noise=true , address=145.79.227.9:4367 , piece_id=3GZSZLVA3BQMIUPSYD56GF7G3UDIFEBJUYEGTXHPTS3WBQYEY5GQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.557313197 (627h8m19.428396232s) [noise=true , address=45.91.116.4:4939 , piece_id=VNDC3YZO4SLOAHO5MKGQJFEDSR4TDHV2Z6YABUAPZE3FBOEJZWSQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.557432204 (627h8m19.428515239s) [noise=true , address=45.91.116.5:4299 , piece_id=OOTXWS7I2QABUC3MLTNZNBNQNPZKB4VZOSVL5IQI5CLSDACVVAGA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.558589517 (627h8m19.429672552s) [noise=true , address=145.79.227.11:4730 , piece_id=WISR335RWCTCOYFIHWRSKDCDTGGCQRCR5OBYJHV5ZAAXU2ZTIJIA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.559259333 (627h8m19.430342368s) [noise=true , address=145.79.227.33:4869 , piece_id=BEZARUUS5WARJNMFI2K3FJC2BJ374KD6Z563R2SBMAWIG23VR5NA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:21.246090342 (627h8m21.117173387s) [upload=549537]: Next returned nil segment
Dec 31 18:37:21.246110060 (627h8m21.117193105s) [upload=549537]: Waiting for error group managing segments...
Dec 31 18:37:21.526859025 (627h8m21.397942071s) [noise=true , address=145.79.227.10:4401 , piece_id=ZQVJJBJTT6RMZMKLLDHF3PH2S2VJN2X4Z263OEPUHE5SFH2EM4FQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.536490225 (627h8m21.40757326s) [noise=true , address=145.79.227.52:4445 , piece_id=B77YXBCHOAQQNXJVWNSRDADDSBVWTM3QLUD5UA74L3BQ5SVG65MQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.537390770 (627h8m21.408473805s) [noise=true , address=145.79.227.2:4115 , piece_id=IHMXPIJARQNI2H3IFXRNEVO33ZIU3FEQRH6NGJRXVV6LS2BUKN3Q, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.547254842 (627h8m21.418337887s) [noise=true , address=145.79.227.13:4802 , piece_id=QTFTLMTDUE2MZNH2Q546WZN53TCMSU6SDGHAYRGXC62RYUQ4JTIQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.561016606 (627h8m21.432099651s) [noise=true , address=145.79.227.9:4367 , piece_id=3GZSZLVA3BQMIUPSYD56GF7G3UDIFEBJUYEGTXHPTS3WBQYEY5GQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.567974195 (627h8m21.43905724s) [noise=true , address=145.79.227.26:4654 , piece_id=JZMRX6LFDNPJW7R4KVBFHGONHU3NFEEZ5GSM7NRJU5LHDSR2QG2Q, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.571333481 (627h8m21.442416516s) [noise=true , address=145.79.227.16:4910 , piece_id=QDARBVNNNM6CTHWOSI2BWT4LVMT5XDGHBILWYNGWNHUASN553AUA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.701924975 (627h8m21.57300802s) [noise=true , address=145.79.227.11:4730 , piece_id=WISR335RWCTCOYFIHWRSKDCDTGGCQRCR5OBYJHV5ZAAXU2ZTIJIA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.819851494 (627h8m21.690934529s) [noise=true , address=77.90.31.11:4553 , piece_id=ZS7BNAXMXUWHOAOJTPBTHAHF6UK6CEMK4DLLJ5N6LYMRE7G7CXSA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.840948916 (627h8m21.712031951s) [noise=true , address=77.90.31.21:4878 , piece_id=PNGE43HPGT44V3HESSNNTXUS2WF3WDVSI23S4QF65NRT2YSNQIOA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.878060091 (627h8m21.749143136s) [noise=true , address=45.91.116.4:4939 , piece_id=VNDC3YZO4SLOAHO5MKGQJFEDSR4TDHV2Z6YABUAPZE3FBOEJZWSQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.974439350 (627h8m21.845522395s) [noise=true , address=45.91.116.5:4299 , piece_id=OOTXWS7I2QABUC3MLTNZNBNQNPZKB4VZOSVL5IQI5CLSDACVVAGA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.999636789 (627h8m21.870719824s) [noise=true , address=45.91.116.10:4083 , piece_id=AYWW2GAOUL64XSPDBR7N2ALI5AVAUXBZBEMUC7FTGMBQGPGUMGYA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.999658621 (627h8m21.870741656s) [seg_pos={0 0}, upload=549537]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:37:21.999841809 (627h8m21.870924854s) [noise=true , address=45.91.116.11:4120 , piece_id=VJK6C2CN7QYB62LMEPYCMYJVAQ72SVOSTP3B7NJX3XBX4OLI3XYQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:21.999850315 (627h8m21.87093338s) [noise=true , address=45.91.116.11:4120 , piece_id=VJK6C2CN7QYB62LMEPYCMYJVAQ72SVOSTP3B7NJX3XBX4OLI3XYQ, seg_pos={0 0}, upload=549537]: Long tail cancellation due to optimal threshold
Dec 31 18:37:22.000200372 (627h8m21.871283427s) [noise=true , address=145.79.227.33:4869 , piece_id=BEZARUUS5WARJNMFI2K3FJC2BJ374KD6Z563R2SBMAWIG23VR5NA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:22.000208077 (627h8m21.871291122s) [noise=true , address=145.79.227.33:4869 , piece_id=BEZARUUS5WARJNMFI2K3FJC2BJ374KD6Z563R2SBMAWIG23VR5NA, seg_pos={0 0}, upload=549537]: Long tail cancellation due to optimal threshold
Dec 31 18:37:22.000375746 (627h8m21.871458802s) [noise=true , address=145.79.227.32:4836 , piece_id=6BG4P7IQ7DP7LK45ULWU7LT3RAHXP47F3VG6AB2VVQATIECJBUMQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:22.000383060 (627h8m21.871466105s) [noise=true , address=145.79.227.32:4836 , piece_id=6BG4P7IQ7DP7LK45ULWU7LT3RAHXP47F3VG6AB2VVQATIECJBUMQ, seg_pos={0 0}, upload=549537]: Long tail cancellation due to optimal threshold
Dec 31 18:37:22.000399561 (627h8m21.871482596s) [seg_pos={0 0}, upload=549537]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:37:22.000474874 (627h8m21.871557909s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:22.000479624 (627h8m21.871562659s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:22.308546746 (627h8m22.179629791s) [upload=549537]: Done uploading segments...
Dec 31 18:37:22.308571582 (627h8m22.179654627s) [upload=549537]: Upload finished. err: <nil>
Duration: 5.844861505s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..........X...X.X...................X.......................] duration:
3.427566957s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:39:18 UTC)
orbiter warning: context deadline exceeded
Duration: 3.427591244s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
5.583451731s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:38:56 UTC)
orbiter warning: context deadline exceeded
Duration: 5.583480907s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
38.405077925s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:38:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 38.405095188s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...........X...X.X...................X......................] duration:
3.428997378s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:38:41 UTC)
orbiter warning: context deadline exceeded
Duration: 3.42902956s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [..X..XXXX..X..XXX....X.X.X.X.....XXX.X.X.X.X.X....XX.X....X.] duration:
6.064700465s trace:
https://jaeger.dev.storj.io/trace/543106d0f5f2249a
FAIL (2025-12-31 18:38:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 18:37:19.295757256 (627h8m19.166840291s) [upload=549537]: Starting upload
Dec 31 18:37:19.295850393 (627h8m19.166933438s) [upload=549537]: Done starting upload
Dec 31 18:37:19.295931367 (627h8m19.167014422s) [upload=549537]: Uploading segments...
Dec 31 18:37:19.296106891 (627h8m19.167189937s) [upload=549537]: Got next segment. Inline: false
Dec 31 18:37:19.296173138 (627h8m19.167256183s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:37:19.296179520 (627h8m19.167262565s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:37:19.546762337 (627h8m19.417845372s) [seg_pos={0 0}, upload=549537]: Begin upload segment...
Dec 31 18:37:19.546826458 (627h8m19.417909494s) [seg_pos={0 0}, upload=549537]: Done begin upload segment.
Dec 31 18:37:19.546990982 (627h8m19.418074017s) [noise=true , address=145.79.227.32:4836 , piece_id=6BG4P7IQ7DP7LK45ULWU7LT3RAHXP47F3VG6AB2VVQATIECJBUMQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.547127642 (627h8m19.418210697s) [noise=true , address=45.91.116.11:4120 , piece_id=VJK6C2CN7QYB62LMEPYCMYJVAQ72SVOSTP3B7NJX3XBX4OLI3XYQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.547652582 (627h8m19.418735627s) [noise=true , address=45.91.116.10:4083 , piece_id=AYWW2GAOUL64XSPDBR7N2ALI5AVAUXBZBEMUC7FTGMBQGPGUMGYA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.548335392 (627h8m19.419418427s) [noise=true , address=77.90.31.21:4878 , piece_id=PNGE43HPGT44V3HESSNNTXUS2WF3WDVSI23S4QF65NRT2YSNQIOA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.548843209 (627h8m19.419926254s) [noise=true , address=145.79.227.13:4802 , piece_id=QTFTLMTDUE2MZNH2Q546WZN53TCMSU6SDGHAYRGXC62RYUQ4JTIQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.549465824 (627h8m19.42054886s) [noise=true , address=145.79.227.52:4445 , piece_id=B77YXBCHOAQQNXJVWNSRDADDSBVWTM3QLUD5UA74L3BQ5SVG65MQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.550758155 (627h8m19.42184119s) [noise=true , address=145.79.227.26:4654 , piece_id=JZMRX6LFDNPJW7R4KVBFHGONHU3NFEEZ5GSM7NRJU5LHDSR2QG2Q, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.552324317 (627h8m19.423407352s) [noise=true , address=77.90.31.11:4553 , piece_id=ZS7BNAXMXUWHOAOJTPBTHAHF6UK6CEMK4DLLJ5N6LYMRE7G7CXSA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.553754501 (627h8m19.424837546s) [noise=true , address=145.79.227.2:4115 , piece_id=IHMXPIJARQNI2H3IFXRNEVO33ZIU3FEQRH6NGJRXVV6LS2BUKN3Q, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.554549765 (627h8m19.42563281s) [noise=true , address=145.79.227.16:4910 , piece_id=QDARBVNNNM6CTHWOSI2BWT4LVMT5XDGHBILWYNGWNHUASN553AUA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.555532045 (627h8m19.42661508s) [noise=true , address=145.79.227.10:4401 , piece_id=ZQVJJBJTT6RMZMKLLDHF3PH2S2VJN2X4Z263OEPUHE5SFH2EM4FQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.555982804 (627h8m19.427065839s) [noise=true , address=145.79.227.9:4367 , piece_id=3GZSZLVA3BQMIUPSYD56GF7G3UDIFEBJUYEGTXHPTS3WBQYEY5GQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.557313197 (627h8m19.428396232s) [noise=true , address=45.91.116.4:4939 , piece_id=VNDC3YZO4SLOAHO5MKGQJFEDSR4TDHV2Z6YABUAPZE3FBOEJZWSQ, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.557432204 (627h8m19.428515239s) [noise=true , address=45.91.116.5:4299 , piece_id=OOTXWS7I2QABUC3MLTNZNBNQNPZKB4VZOSVL5IQI5CLSDACVVAGA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.558589517 (627h8m19.429672552s) [noise=true , address=145.79.227.11:4730 , piece_id=WISR335RWCTCOYFIHWRSKDCDTGGCQRCR5OBYJHV5ZAAXU2ZTIJIA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:19.559259333 (627h8m19.430342368s) [noise=true , address=145.79.227.33:4869 , piece_id=BEZARUUS5WARJNMFI2K3FJC2BJ374KD6Z563R2SBMAWIG23VR5NA, seg_pos={0 0}, upload=549537]: Uploading piece...
Dec 31 18:37:21.246090342 (627h8m21.117173387s) [upload=549537]: Next returned nil segment
Dec 31 18:37:21.246110060 (627h8m21.117193105s) [upload=549537]: Waiting for error group managing segments...
Dec 31 18:37:21.526859025 (627h8m21.397942071s) [noise=true , address=145.79.227.10:4401 , piece_id=ZQVJJBJTT6RMZMKLLDHF3PH2S2VJN2X4Z263OEPUHE5SFH2EM4FQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.536490225 (627h8m21.40757326s) [noise=true , address=145.79.227.52:4445 , piece_id=B77YXBCHOAQQNXJVWNSRDADDSBVWTM3QLUD5UA74L3BQ5SVG65MQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.537390770 (627h8m21.408473805s) [noise=true , address=145.79.227.2:4115 , piece_id=IHMXPIJARQNI2H3IFXRNEVO33ZIU3FEQRH6NGJRXVV6LS2BUKN3Q, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.547254842 (627h8m21.418337887s) [noise=true , address=145.79.227.13:4802 , piece_id=QTFTLMTDUE2MZNH2Q546WZN53TCMSU6SDGHAYRGXC62RYUQ4JTIQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.561016606 (627h8m21.432099651s) [noise=true , address=145.79.227.9:4367 , piece_id=3GZSZLVA3BQMIUPSYD56GF7G3UDIFEBJUYEGTXHPTS3WBQYEY5GQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.567974195 (627h8m21.43905724s) [noise=true , address=145.79.227.26:4654 , piece_id=JZMRX6LFDNPJW7R4KVBFHGONHU3NFEEZ5GSM7NRJU5LHDSR2QG2Q, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.571333481 (627h8m21.442416516s) [noise=true , address=145.79.227.16:4910 , piece_id=QDARBVNNNM6CTHWOSI2BWT4LVMT5XDGHBILWYNGWNHUASN553AUA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.701924975 (627h8m21.57300802s) [noise=true , address=145.79.227.11:4730 , piece_id=WISR335RWCTCOYFIHWRSKDCDTGGCQRCR5OBYJHV5ZAAXU2ZTIJIA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.819851494 (627h8m21.690934529s) [noise=true , address=77.90.31.11:4553 , piece_id=ZS7BNAXMXUWHOAOJTPBTHAHF6UK6CEMK4DLLJ5N6LYMRE7G7CXSA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.840948916 (627h8m21.712031951s) [noise=true , address=77.90.31.21:4878 , piece_id=PNGE43HPGT44V3HESSNNTXUS2WF3WDVSI23S4QF65NRT2YSNQIOA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.878060091 (627h8m21.749143136s) [noise=true , address=45.91.116.4:4939 , piece_id=VNDC3YZO4SLOAHO5MKGQJFEDSR4TDHV2Z6YABUAPZE3FBOEJZWSQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.974439350 (627h8m21.845522395s) [noise=true , address=45.91.116.5:4299 , piece_id=OOTXWS7I2QABUC3MLTNZNBNQNPZKB4VZOSVL5IQI5CLSDACVVAGA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.999636789 (627h8m21.870719824s) [noise=true , address=45.91.116.10:4083 , piece_id=AYWW2GAOUL64XSPDBR7N2ALI5AVAUXBZBEMUC7FTGMBQGPGUMGYA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: <nil>
Dec 31 18:37:21.999658621 (627h8m21.870741656s) [seg_pos={0 0}, upload=549537]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:37:21.999841809 (627h8m21.870924854s) [noise=true , address=45.91.116.11:4120 , piece_id=VJK6C2CN7QYB62LMEPYCMYJVAQ72SVOSTP3B7NJX3XBX4OLI3XYQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:21.999850315 (627h8m21.87093338s) [noise=true , address=45.91.116.11:4120 , piece_id=VJK6C2CN7QYB62LMEPYCMYJVAQ72SVOSTP3B7NJX3XBX4OLI3XYQ, seg_pos={0 0}, upload=549537]: Long tail cancellation due to optimal threshold
Dec 31 18:37:22.000200372 (627h8m21.871283427s) [noise=true , address=145.79.227.33:4869 , piece_id=BEZARUUS5WARJNMFI2K3FJC2BJ374KD6Z563R2SBMAWIG23VR5NA, seg_pos={0 0}, upload=549537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:22.000208077 (627h8m21.871291122s) [noise=true , address=145.79.227.33:4869 , piece_id=BEZARUUS5WARJNMFI2K3FJC2BJ374KD6Z563R2SBMAWIG23VR5NA, seg_pos={0 0}, upload=549537]: Long tail cancellation due to optimal threshold
Dec 31 18:37:22.000375746 (627h8m21.871458802s) [noise=true , address=145.79.227.32:4836 , piece_id=6BG4P7IQ7DP7LK45ULWU7LT3RAHXP47F3VG6AB2VVQATIECJBUMQ, seg_pos={0 0}, upload=549537]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:22.000383060 (627h8m21.871466105s) [noise=true , address=145.79.227.32:4836 , piece_id=6BG4P7IQ7DP7LK45ULWU7LT3RAHXP47F3VG6AB2VVQATIECJBUMQ, seg_pos={0 0}, upload=549537]: Long tail cancellation due to optimal threshold
Dec 31 18:37:22.000399561 (627h8m21.871482596s) [seg_pos={0 0}, upload=549537]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:37:22.000474874 (627h8m21.871557909s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:22.000479624 (627h8m21.871562659s) [upload=549537]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:22.308546746 (627h8m22.179629791s) [upload=549537]: Done uploading segments...
Dec 31 18:37:22.308571582 (627h8m22.179654627s) [upload=549537]: Upload finished. err: <nil>
Duration: 6.064721425s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.003757645s trace:
https://jaeger.dev.storj.io/trace/24432a4fc7a385da
FAIL (2025-12-31 18:38:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648107778 (627h8m3.519190823s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648486920 (627h8m3.519569965s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.648843990 (627h8m3.519927035s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649285190 (627h8m3.520368235s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649634716 (627h8m3.520717751s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.649989111 (627h8m3.521072156s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650347623 (627h8m3.521430668s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.650715394 (627h8m3.521798439s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.651585130 (627h8m3.522668175s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652042801 (627h8m3.523125836s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652460777 (627h8m3.523543822s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.652926143 (627h8m3.524009178s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.654529877 (627h8m3.525612912s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.656264951 (627h8m3.527347986s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.659324617 (627h8m3.530407662s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.662463203 (627h8m3.533546248s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:03.665326735 (627h8m3.536409781s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Uploading piece...
Dec 31 18:37:05.121169544 (627h8m4.992252619s) [upload=549533]: Next returned nil segment
Dec 31 18:37:05.121186426 (627h8m4.992269481s) [upload=549533]: Waiting for error group managing segments...
Dec 31 18:37:05.493475557 (627h8m5.364558592s) [noise=true , address=64.49.13.138:4490 , piece_id=LSEVOGUTOL4FKEHQS6H542XTKJNMGX4WULOHPCPXPJ5SNEBQU7KA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.602994467 (627h8m5.474077512s) [noise=true , address=145.79.227.14:4858 , piece_id=E7M3BTU56RYW6POPI7T35IFD5DSE2VNVHZHOYTLBHRPJJU4GHRGA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.627459732 (627h8m5.498542787s) [noise=true , address=145.79.227.43:4166 , piece_id=3ODERP44NMR3VV2SHP2CXSGKWH64Y2CH6G3YOT3QYEDZDUYMAGVA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.637909839 (627h8m5.508992884s) [noise=true , address=145.79.227.35:4515 , piece_id=ZEJ3RLBEIQVCBVJL56ODMLYAXMZXENJ6CB5ZZRTBW6HEHFOG2TMQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.683777739 (627h8m5.554860774s) [noise=true , address=145.79.227.6:4287 , piece_id=CM67SR4I6MDYU56JXH5NTTFQGRUFYJAZC27YT2XPXUULEZDNUSEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.711588996 (627h8m5.582672041s) [noise=true , address=77.90.31.17:4802 , piece_id=RSQ3POHH7DBIFS3ZPJZ7HNW4K7ERCMDTF7KC63KPROWZZLZ7JIJQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.723273444 (627h8m5.594356479s) [noise=true , address=77.90.31.12:4611 , piece_id=ZAXXFHOVYRAEIKXXXTJYDJC3SDBOH3GSEGHYGKMHK3MM42I5TPYA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.838882139 (627h8m5.709965184s) [noise=true , address=77.90.31.11:4559 , piece_id=I6SU3ZAMV723V4RTMSJKRFZY35KCTP3TTOHL7UAKPR6REJKWTZWQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.846600877 (627h8m5.717683922s) [noise=true , address=45.91.116.14:4195 , piece_id=SGXWVXANKMETFZHUE3WI2FJI55JLTXF2SVV2D2Y7GXWNOBCWQCNA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:05.858828230 (627h8m5.729911275s) [noise=true , address=45.91.116.11:4139 , piece_id=I7EU6LAQ72C3P5TILBLKRGGWDN2RBHMIBXTQYVUFC7ZSBICSC3UQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.391976308 (627h8m6.263059343s) [noise=true , address=145.79.227.30:4778 , piece_id=HCKVT7GUWTA3F35TKNSE5CGVEIPBVV63MYVHJKQPY7TFECYRC7EQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.451470513 (627h8m6.322553549s) [noise=true , address=193.149.164.143:4696 , piece_id=DF3WJ2BLW6RX3OEAOULAXB2375ZUYNM6KM6ARJFD4U7OBKHVBOEA, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.477640595 (627h8m6.34872363s) [noise=true , address=145.79.227.40:4055 , piece_id=WB6BJIBSCF7CTGF3JVDVZPPRHIM4ZHZP4BPDQEFKVFSIZOSM2XOQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483491506 (627h8m6.354574551s) [noise=true , address=145.79.227.34:4925 , piece_id=NH4WRJBOJKCRZYCH2H23VCEYVPXPTLB3HBCCGN6L5U5K3P77KPFQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: <nil>
Dec 31 18:37:06.483516744 (627h8m6.35459979s) [seg_pos={0 0}, upload=549533]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:37:06.483734759 (627h8m6.354817804s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RZEXg8h3qbrw3mJadmpudtBsVadHMb4knwtueL7GhdHLkESYC): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.483743796 (627h8m6.354826841s) [noise=true , address=145.79.227.33:4876 , piece_id=2LA6NHTCMVMAY4LHIL3DSCX4OXHWY2WMG7J2NMMPBO7PEVVEOBBQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484054658 (627h8m6.355137703s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484063545 (627h8m6.35514659s) [noise=true , address=77.90.31.18:4840 , piece_id=YS3MAD3V4BGSQENVDOKLG334NBQDF4JG2OLYMK2Q6QJKUH4XPOPQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484205926 (627h8m6.355288961s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YgFc7P9RCbYw9u2wfow8Z3DMwFvJ6AnCsvzoxU9pTwPWGw8MV): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484213520 (627h8m6.355296565s) [noise=true , address=145.79.227.49:4359 , piece_id=JVWHFG6TDDJZ4EY2E6HSWKHDSA3REXHQ4ULAEMHA2KPJ73X5WMUQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484359078 (627h8m6.355442123s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:118qGQLFSNbsYsvFrKjTXXWbFYu2sRaRHrUNkKNCrNVpzVQRtG): protocol: expected piece hash; context canceled; EOF
Dec 31 18:37:06.484367073 (627h8m6.355450118s) [noise=true , address=145.79.227.33:4885 , piece_id=W7IMHDBWRQLQVDLYDGVNAPXUUDC2BVXHHMTGB5AKUGMB5P2WQITQ, seg_pos={0 0}, upload=549533]: Long tail cancellation due to optimal threshold
Dec 31 18:37:06.484387882 (627h8m6.355470917s) [seg_pos={0 0}, upload=549533]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 18:37:06.484470069 (627h8m6.355553104s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:37:06.484476792 (627h8m6.355559837s) [upload=549533]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:37:06.808174710 (627h8m6.679257755s) [upload=549533]: Done uploading segments...
Dec 31 18:37:06.808200198 (627h8m6.679283233s) [upload=549533]: Upload finished. err: <nil>
Duration: 1m0.003822829s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............X...X.X...................X.....................] duration:
3.451794771s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:37:58 UTC)
orbiter warning: context deadline exceeded
Duration: 3.451821501s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
6.326582506s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:37:50 UTC)
orbiter warning: context deadline exceeded
Duration: 6.326610349s
edge
gateway
us-select-1
|