Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.............X...........X...X....X....X.XX...........X.....] duration:
14.964680952s trace:
https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd
FAIL (2025-12-29 14:14:54 UTC)
orbiter warning: context deadline exceeded
Duration: 14.964707794s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [....................................XXX..X..XX..............] duration:
15.582645605s trace:
https://jaeger.dev.storj.io/trace/266137264da4c90
FAIL (2025-12-29 14:14:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded
======LOGS======
NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment
Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments...
Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection
Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection
Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection
Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments...
Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil>
Duration: 15.582671435s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
36 out of 60 is failed (threshold=3) [..X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.XX] duration:
30.003286178s trace:
https://jaeger.dev.storj.io/trace/7e0577d02dabde35
FAIL (2025-12-29 14:14:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: VDCFQPKRBRGJYRQCRHTRLTK5XQD6NKBF4NZ3XALABZH6CSSQYP2A) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: DYCNFQO4LQSIVGXTVBMGUZ2CMOWNA2WDWDFKUT7FRPEVHJTV6MLQ) context deadline exceeded
======LOGS======
9 14:14:12.163507032 (574h45m12.034590067s) [upload=503862]: Starting upload
Dec 29 14:14:12.163609357 (574h45m12.034692392s) [upload=503862]: Done starting upload
Dec 29 14:14:12.163640246 (574h45m12.034723281s) [upload=503862]: Uploading segments...
Dec 29 14:14:12.163812053 (574h45m12.034895088s) [upload=503862]: Got next segment. Inline: false
Dec 29 14:14:12.163840867 (574h45m12.034923902s) [upload=503862]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 14:14:12.163845326 (574h45m12.034928371s) [upload=503862]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 14:14:12.414949035 (574h45m12.28603207s) [seg_pos={0 0}, upload=503862]: Begin upload segment...
Dec 29 14:14:12.414997457 (574h45m12.286080492s) [seg_pos={0 0}, upload=503862]: Done begin upload segment.
Dec 29 14:14:12.415034197 (574h45m12.286117232s) [noise=true , address=145.79.227.39:4014 , piece_id=CKZPX6CLYEGNJ7DKZPKUILJKK3MIGQRVNWQ4GPHM7R62YFVWNNAA, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.415055738 (574h45m12.286138773s) [noise=true , address=145.79.227.15:4874 , piece_id=WSF6Q4IHGSA66NZZE5LC5ID5MJUCSZBBCPLBEHZI7IB5GT6QVIQQ, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.415617848 (574h45m12.286700894s) [noise=true , address=45.91.116.8:4047 , piece_id=C5VUNQBNNF3RRNAI7SXRYU2UG4HFWLHYIWNJV3YOHWHIKQSSAGMQ, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.416044621 (574h45m12.287127656s) [noise=true , address=145.79.227.16:4911 , piece_id=6VZR474ISTRJY2S4OA325OCA6KRI3TQPU6NM57HTBIV6NQDO7BYQ, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.416580422 (574h45m12.287663457s) [noise=true , address=45.91.116.11:4119 , piece_id=OQP7LRJH2U7DUCVO56SVPQCEYGRXCJ7LJUAEKBKMCFECNO4M2H5A, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.417150377 (574h45m12.288233422s) [noise=true , address=45.91.116.13:4156 , piece_id=VDCFQPKRBRGJYRQCRHTRLTK5XQD6NKBF4NZ3XALABZH6CSSQYP2A, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.417246700 (574h45m12.288329745s) [noise=true , address=145.79.227.4:4185 , piece_id=ZXHM53FY4HDOVOLUFBM6H2UODLS6OB73BA3Y52TS5UVMGPKELGNQ, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.417918870 (574h45m12.289001915s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYX3VI65DTHNNQ7NJAVRPUJRRHLTKVRHU4LLBLPWH2KKU7UNO4A, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.418462244 (574h45m12.289545289s) [noise=true , address=145.79.227.43:4157 , piece_id=VTZZEJVYIFKCSN5VDW55ZR3YH5KO3B3UJVVMFVZVFZY2T4CNCBRA, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.419096833 (574h45m12.290179888s) [noise=true , address=145.79.227.32:4834 , piece_id=MKNCXCYM5RTILTYKIEHTSGYS2XR3TKV6OFEEENXZYZRUFO7ELUOA, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.419447310 (574h45m12.290530356s) [noise=true , address=145.79.227.36:4517 , piece_id=ZPGXWIPY6CUZGK3ESJKNS4EOPX4IAQKMQ26UCE3XURSPRL3D243Q, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.419788539 (574h45m12.290871584s) [noise=true , address=145.79.227.13:4804 , piece_id=RQ52HBHMIELRZV6UXCJWALONJ5KIQUE6RPKT7ZP4EJCAEDPX4VPA, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.420094913 (574h45m12.291177958s) [noise=true , address=145.79.227.7:4294 , piece_id=3OEY5YLM5M5M56LCBWQIWCFT3JZ7MUDLLJALKMTJVXXLDN7LL4QA, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.420397950 (574h45m12.291480995s) [noise=true , address=77.90.31.16:4736 , piece_id=MWHE2ORCE24CPVY6EJZHZT4ZOFBPOZ2OIPCM6UNGVHVZI562UA3Q, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.420696248 (574h45m12.291779283s) [noise=true , address=145.79.227.17:4972 , piece_id=XNLF3UWDI5PNEZWWCW5OEAPKMAM2IQYQWTXJOBV4GDC5LEI6Z34A, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:12.422826905 (574h45m12.29390995s) [noise=true , address=45.91.116.14:4192 , piece_id=DYCNFQO4LQSIVGXTVBMGUZ2CMOWNA2WDWDFKUT7FRPEVHJTV6MLQ, seg_pos={0 0}, upload=503862]: Uploading piece...
Dec 29 14:14:14.114724822 (574h45m13.985807867s) [upload=503862]: Next returned nil segment
Dec 29 14:14:14.114739259 (574h45m13.985822314s) [upload=503862]: Waiting for error group managing segments...
Dec 29 14:14:14.398168430 (574h45m14.269251475s) [noise=true , address=145.79.227.13:4804 , piece_id=RQ52HBHMIELRZV6UXCJWALONJ5KIQUE6RPKT7ZP4EJCAEDPX4VPA, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.398517314 (574h45m14.269600349s) [noise=true , address=145.79.227.16:4911 , piece_id=6VZR474ISTRJY2S4OA325OCA6KRI3TQPU6NM57HTBIV6NQDO7BYQ, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.400968031 (574h45m14.272051066s) [noise=true , address=145.79.227.4:4185 , piece_id=ZXHM53FY4HDOVOLUFBM6H2UODLS6OB73BA3Y52TS5UVMGPKELGNQ, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.405467489 (574h45m14.276550534s) [noise=true , address=145.79.227.39:4014 , piece_id=CKZPX6CLYEGNJ7DKZPKUILJKK3MIGQRVNWQ4GPHM7R62YFVWNNAA, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.408636944 (574h45m14.279719989s) [noise=true , address=145.79.227.36:4517 , piece_id=ZPGXWIPY6CUZGK3ESJKNS4EOPX4IAQKMQ26UCE3XURSPRL3D243Q, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.408909173 (574h45m14.279992218s) [noise=true , address=145.79.227.32:4834 , piece_id=MKNCXCYM5RTILTYKIEHTSGYS2XR3TKV6OFEEENXZYZRUFO7ELUOA, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.414730758 (574h45m14.285813803s) [noise=true , address=145.79.227.15:4874 , piece_id=WSF6Q4IHGSA66NZZE5LC5ID5MJUCSZBBCPLBEHZI7IB5GT6QVIQQ, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.425850942 (574h45m14.296933987s) [noise=true , address=145.79.227.12:4767 , piece_id=RTYX3VI65DTHNNQ7NJAVRPUJRRHLTKVRHU4LLBLPWH2KKU7UNO4A, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.432529619 (574h45m14.303612664s) [noise=true , address=145.79.227.17:4972 , piece_id=XNLF3UWDI5PNEZWWCW5OEAPKMAM2IQYQWTXJOBV4GDC5LEI6Z34A, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.438269518 (574h45m14.309352563s) [noise=true , address=145.79.227.7:4294 , piece_id=3OEY5YLM5M5M56LCBWQIWCFT3JZ7MUDLLJALKMTJVXXLDN7LL4QA, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.689112511 (574h45m14.560195556s) [noise=true , address=45.91.116.13:4156 , piece_id=VDCFQPKRBRGJYRQCRHTRLTK5XQD6NKBF4NZ3XALABZH6CSSQYP2A, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:14.823251632 (574h45m14.694334667s) [noise=true , address=45.91.116.14:4192 , piece_id=DYCNFQO4LQSIVGXTVBMGUZ2CMOWNA2WDWDFKUT7FRPEVHJTV6MLQ, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:15.554065621 (574h45m15.425148656s) [noise=true , address=45.91.116.11:4119 , piece_id=OQP7LRJH2U7DUCVO56SVPQCEYGRXCJ7LJUAEKBKMCFECNO4M2H5A, seg_pos={0 0}, upload=503862]: Done uploading piece. err: <nil>
Dec 29 14:14:15.554089526 (574h45m15.425172571s) [seg_pos={0 0}, upload=503862]: Segment reached optimal threshold of 13 pieces.
Dec 29 14:14:15.554334012 (574h45m15.425417067s) [noise=true , address=45.91.116.8:4047 , piece_id=C5VUNQBNNF3RRNAI7SXRYU2UG4HFWLHYIWNJV3YOHWHIKQSSAGMQ, seg_pos={0 0}, upload=503862]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 29 14:14:15.554343339 (574h45m15.425426374s) [noise=true , address=45.91.116.8:4047 , piece_id=C5VUNQBNNF3RRNAI7SXRYU2UG4HFWLHYIWNJV3YOHWHIKQSSAGMQ, seg_pos={0 0}, upload=503862]: Long tail cancellation due to optimal threshold
Dec 29 14:14:15.554523633 (574h45m15.425606678s) [noise=true , address=77.90.31.16:4736 , piece_id=MWHE2ORCE24CPVY6EJZHZT4ZOFBPOZ2OIPCM6UNGVHVZI562UA3Q, seg_pos={0 0}, upload=503862]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:37704->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:37704->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37704->77.90.31.16:4736: use of closed network connection
Dec 29 14:14:15.554532649 (574h45m15.425615684s) [noise=true , address=77.90.31.16:4736 , piece_id=MWHE2ORCE24CPVY6EJZHZT4ZOFBPOZ2OIPCM6UNGVHVZI562UA3Q, seg_pos={0 0}, upload=503862]: Long tail cancellation due to optimal threshold
Dec 29 14:14:15.554636948 (574h45m15.425720023s) [noise=true , address=145.79.227.43:4157 , piece_id=VTZZEJVYIFKCSN5VDW55ZR3YH5KO3B3UJVVMFVZVFZY2T4CNCBRA, seg_pos={0 0}, upload=503862]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF
Dec 29 14:14:15.554662958 (574h45m15.425746003s) [noise=true , address=145.79.227.43:4157 , piece_id=VTZZEJVYIFKCSN5VDW55ZR3YH5KO3B3UJVVMFVZVFZY2T4CNCBRA, seg_pos={0 0}, upload=503862]: Long tail cancellation due to optimal threshold
Dec 29 14:14:15.554700489 (574h45m15.425783524s) [seg_pos={0 0}, upload=503862]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 14:14:15.554839173 (574h45m15.425922218s) [upload=503862]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:14:15.554848861 (574h45m15.425931937s) [upload=503862]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:14:15.856340006 (574h45m15.727423061s) [upload=503862]: Done uploading segments...
Dec 29 14:14:15.856362860 (574h45m15.727445905s) [upload=503862]: Upload finished. err: <nil>
Duration: 30.003311095s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..............X...........X...X....X....X.XX...........X....] duration:
52.842781381s trace:
https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd
FAIL (2025-12-29 14:14:33 UTC)
orbiter warning: context deadline exceeded
Duration: 52.842808814s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [..............X...X.X.XX....X........X.X............X..X.XX.] duration:
21.930673645s trace:
https://jaeger.dev.storj.io/trace/2933e45b57549e03
FAIL (2025-12-29 14:14:32 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 21.930702951s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....................................XXX..X..XX.............] duration:
53.755477493s trace:
https://jaeger.dev.storj.io/trace/266137264da4c90
FAIL (2025-12-29 14:14:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded
======LOGS======
NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment
Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments...
Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection
Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection
Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection
Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments...
Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil>
Duration: 53.755495668s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXXX] duration:
5.003487775s trace:
https://jaeger.dev.storj.io/trace/513565a8dd354d42
FAIL (2025-12-29 14:14:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.003513454s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [...............X...X.X.XX....X........X.X............X..X.XX] duration:
30.001626263s trace:
https://jaeger.dev.storj.io/trace/2933e45b57549e03
FAIL (2025-12-29 14:14:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001639828s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
35 out of 60 is failed (threshold=3) [...X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.X] duration:
30.002704028s trace:
https://jaeger.dev.storj.io/trace/6d873025632bc8c3
FAIL (2025-12-29 14:13:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 14:13:07.523114686 (574h44m7.394197721s) [upload=503849]: Starting upload
Dec 29 14:13:07.523217041 (574h44m7.394300086s) [upload=503849]: Done starting upload
Dec 29 14:13:07.523268439 (574h44m7.394351484s) [upload=503849]: Uploading segments...
Dec 29 14:13:07.523479721 (574h44m7.394562766s) [upload=503849]: Got next segment. Inline: false
Dec 29 14:13:07.523522583 (574h44m7.394605618s) [upload=503849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 14:13:07.523528724 (574h44m7.394611769s) [upload=503849]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 14:13:07.775722017 (574h44m7.646805062s) [seg_pos={0 0}, upload=503849]: Begin upload segment...
Dec 29 14:13:07.775775129 (574h44m7.646858174s) [seg_pos={0 0}, upload=503849]: Done begin upload segment.
Dec 29 14:13:07.775810656 (574h44m7.646893701s) [noise=true , address=145.79.227.6:4257 , piece_id=PBOXMNGC64QUJW3K23CKY24MPNEMRV3WKL242WX53LL3LQXQCFNA, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.775834292 (574h44m7.646917327s) [noise=true , address=145.79.227.50:4374 , piece_id=7CCGOAJSPYCWZSQU5FBF7NWOSRLPOMMVEO2HI2I22NPRRO74KMTA, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.776277034 (574h44m7.647360079s) [noise=true , address=145.79.227.16:4909 , piece_id=YVD4ECKAGAVAHKTOFHYU3WLH6ZQ42MTOI34E3MD6XI2747APEQNQ, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.776679791 (574h44m7.647762836s) [noise=true , address=145.79.227.4:4187 , piece_id=NEOTOXKLB73KTOXARKTV7OVDNYBSSKX2MCVI5RX6TEI6KWDBF5ZQ, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.777118898 (574h44m7.648201933s) [noise=true , address=145.79.227.2:4115 , piece_id=I3VHZYC6V46C4MB4ROI3LE7R45AMFZKUWDNQ7AVBL7IH4AHCWW4Q, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.777520833 (574h44m7.648603868s) [noise=true , address=45.91.116.11:4117 , piece_id=EZJW6VCZ5K5PNLH3BCK4IWDLKYRNYWOV3KLD6JRVX7QDYEYJHRFQ, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.778128360 (574h44m7.649211395s) [noise=true , address=145.79.227.32:4834 , piece_id=YXX7JZJPJSCY2LPQKAQCNYBXER4CQB2S3BPMHKA7H2Y4LYF4MVLQ, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.778580942 (574h44m7.649663977s) [noise=true , address=45.91.116.8:4048 , piece_id=F2E4BTRCU34Y2HZECZYMTKX567EBRN2YS2VTPM7JRJTBOX6K7WPQ, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.779581566 (574h44m7.650664601s) [noise=true , address=145.79.227.13:4803 , piece_id=GWNGLLFDPGOZ6HDMK2TPQOYHQRAJFHR5ETQROO3EF3RSWZMV3WYA, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.780132155 (574h44m7.6512152s) [noise=true , address=77.90.31.11:4555 , piece_id=JY75G4AG4ONK3GR54M6KDA4V3EZWVBWTGYNUBTUNODQS5OSE7LOA, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.781114385 (574h44m7.65219742s) [noise=true , address=45.91.116.12:4225 , piece_id=MODQXNU2DFWPCXSOQGHCXXGLRTD2M7LCKGZTYVJZ7KWSSYVEMEJA, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.782474014 (574h44m7.653557049s) [noise=true , address=145.79.227.11:4730 , piece_id=TBJVYVFRZ5GZXMQOUVT2D525NU65XGY3PZYOJRIMYYIXNJVTK6WA, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.782836094 (574h44m7.653919139s) [noise=true , address=145.79.227.46:4603 , piece_id=UBD77X7R7B35JZY5B7TX5MLLSRHZXGE24QNVDWJTPOE33UB2CXLQ, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.784040617 (574h44m7.655123662s) [noise=true , address=45.91.116.13:4153 , piece_id=DPZVJGXFT72CFAQA4EODT6RMVBEYCBYEWRPVRFTPAZZSPDGZPDMQ, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.785404826 (574h44m7.656487891s) [noise=true , address=145.79.227.51:4410 , piece_id=HAP7DL6REGYTAU6CT4YLO5LUQWBA3T2KJGMDT3CFJQUWHQKVVB6Q, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:07.787258515 (574h44m7.65834155s) [noise=true , address=145.79.227.18:4007 , piece_id=KPZEAGQ5AGCSPDI73747POFW44LOZME6XTT33YWWU7GOIGOE3S5Q, seg_pos={0 0}, upload=503849]: Uploading piece...
Dec 29 14:13:09.480156565 (574h44m9.3512396s) [upload=503849]: Next returned nil segment
Dec 29 14:13:09.480168888 (574h44m9.351251923s) [upload=503849]: Waiting for error group managing segments...
Dec 29 14:13:09.756922829 (574h44m9.628005864s) [noise=true , address=145.79.227.32:4834 , piece_id=YXX7JZJPJSCY2LPQKAQCNYBXER4CQB2S3BPMHKA7H2Y4LYF4MVLQ, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.763343485 (574h44m9.63442652s) [noise=true , address=145.79.227.13:4803 , piece_id=GWNGLLFDPGOZ6HDMK2TPQOYHQRAJFHR5ETQROO3EF3RSWZMV3WYA, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.763387159 (574h44m9.634470204s) [noise=true , address=145.79.227.11:4730 , piece_id=TBJVYVFRZ5GZXMQOUVT2D525NU65XGY3PZYOJRIMYYIXNJVTK6WA, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.766588595 (574h44m9.63767164s) [noise=true , address=145.79.227.2:4115 , piece_id=I3VHZYC6V46C4MB4ROI3LE7R45AMFZKUWDNQ7AVBL7IH4AHCWW4Q, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.770286495 (574h44m9.64136953s) [noise=true , address=145.79.227.18:4007 , piece_id=KPZEAGQ5AGCSPDI73747POFW44LOZME6XTT33YWWU7GOIGOE3S5Q, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.771811789 (574h44m9.642894835s) [noise=true , address=145.79.227.46:4603 , piece_id=UBD77X7R7B35JZY5B7TX5MLLSRHZXGE24QNVDWJTPOE33UB2CXLQ, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.779195399 (574h44m9.650278444s) [noise=true , address=145.79.227.51:4410 , piece_id=HAP7DL6REGYTAU6CT4YLO5LUQWBA3T2KJGMDT3CFJQUWHQKVVB6Q, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.780826936 (574h44m9.651909981s) [noise=true , address=145.79.227.4:4187 , piece_id=NEOTOXKLB73KTOXARKTV7OVDNYBSSKX2MCVI5RX6TEI6KWDBF5ZQ, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.788127728 (574h44m9.659210773s) [noise=true , address=145.79.227.6:4257 , piece_id=PBOXMNGC64QUJW3K23CKY24MPNEMRV3WKL242WX53LL3LQXQCFNA, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.798987697 (574h44m9.670070742s) [noise=true , address=145.79.227.16:4909 , piece_id=YVD4ECKAGAVAHKTOFHYU3WLH6ZQ42MTOI34E3MD6XI2747APEQNQ, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.806986648 (574h44m9.678069683s) [noise=true , address=145.79.227.50:4374 , piece_id=7CCGOAJSPYCWZSQU5FBF7NWOSRLPOMMVEO2HI2I22NPRRO74KMTA, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:09.952068766 (574h44m9.823151811s) [noise=true , address=77.90.31.11:4555 , piece_id=JY75G4AG4ONK3GR54M6KDA4V3EZWVBWTGYNUBTUNODQS5OSE7LOA, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:10.064755909 (574h44m9.935838954s) [noise=true , address=45.91.116.8:4048 , piece_id=F2E4BTRCU34Y2HZECZYMTKX567EBRN2YS2VTPM7JRJTBOX6K7WPQ, seg_pos={0 0}, upload=503849]: Done uploading piece. err: <nil>
Dec 29 14:13:10.064777560 (574h44m9.935860595s) [seg_pos={0 0}, upload=503849]: Segment reached optimal threshold of 13 pieces.
Dec 29 14:13:10.064996177 (574h44m9.936079212s) [noise=true , address=45.91.116.12:4225 , piece_id=MODQXNU2DFWPCXSOQGHCXXGLRTD2M7LCKGZTYVJZ7KWSSYVEMEJA, seg_pos={0 0}, upload=503849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 29 14:13:10.065005995 (574h44m9.93608903s) [noise=true , address=45.91.116.12:4225 , piece_id=MODQXNU2DFWPCXSOQGHCXXGLRTD2M7LCKGZTYVJZ7KWSSYVEMEJA, seg_pos={0 0}, upload=503849]: Long tail cancellation due to optimal threshold
Dec 29 14:13:10.065021274 (574h44m9.936104319s) [noise=true , address=45.91.116.13:4153 , piece_id=DPZVJGXFT72CFAQA4EODT6RMVBEYCBYEWRPVRFTPAZZSPDGZPDMQ, seg_pos={0 0}, upload=503849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:13:10.065031965 (574h44m9.93611501s) [noise=true , address=45.91.116.13:4153 , piece_id=DPZVJGXFT72CFAQA4EODT6RMVBEYCBYEWRPVRFTPAZZSPDGZPDMQ, seg_pos={0 0}, upload=503849]: Long tail cancellation due to optimal threshold
Dec 29 14:13:10.065224070 (574h44m9.936307115s) [noise=true , address=45.91.116.11:4117 , piece_id=EZJW6VCZ5K5PNLH3BCK4IWDLKYRNYWOV3KLD6JRVX7QDYEYJHRFQ, seg_pos={0 0}, upload=503849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 29 14:13:10.065232997 (574h44m9.936316042s) [noise=true , address=45.91.116.11:4117 , piece_id=EZJW6VCZ5K5PNLH3BCK4IWDLKYRNYWOV3KLD6JRVX7QDYEYJHRFQ, seg_pos={0 0}, upload=503849]: Long tail cancellation due to optimal threshold
Dec 29 14:13:10.065254368 (574h44m9.936337413s) [seg_pos={0 0}, upload=503849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 14:13:10.065395236 (574h44m9.936478281s) [upload=503849]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:13:10.065402009 (574h44m9.936485054s) [upload=503849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:13:10.355716512 (574h44m10.226799557s) [upload=503849]: Done uploading segments...
Dec 29 14:13:10.355778581 (574h44m10.226861616s) [upload=503849]: Upload finished. err: <nil>
Duration: 30.002730779s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................................XXX..X..XX............] duration:
46.639781148s trace:
https://jaeger.dev.storj.io/trace/266137264da4c90
FAIL (2025-12-29 14:13:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded
======LOGS======
NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment
Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments...
Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection
Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection
Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection
Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments...
Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil>
Duration: 46.639797849s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [...............X...........X...X....X....X.XX...........X...] duration:
15.150587869s trace:
https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd
FAIL (2025-12-29 14:13:19 UTC)
orbiter warning: context deadline exceeded
Duration: 15.150600405s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [................X...X.X.XX....X........X.X............X..X.X] duration:
30.001779763s trace:
https://jaeger.dev.storj.io/trace/43fad61c54f6bbcb
FAIL (2025-12-29 14:13:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001796085s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXXX] duration:
5.00118536s trace:
https://jaeger.dev.storj.io/trace/188d20428ae7ecd4
FAIL (2025-12-29 14:13:02 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001208885s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [................X...........X...X....X....X.XX...........X..] duration:
24.124284246s trace:
https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd
FAIL (2025-12-29 14:12:49 UTC)
orbiter warning: context deadline exceeded
Duration: 24.124322058s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.......................................XXX..X..XX...........] duration:
50.031214584s trace:
https://jaeger.dev.storj.io/trace/266137264da4c90
FAIL (2025-12-29 14:12:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded
======LOGS======
NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment
Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments...
Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection
Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection
Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection
Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments...
Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil>
Duration: 50.031230755s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
35 out of 60 is failed (threshold=3) [X...X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX.] duration:
7.079235919s trace:
https://jaeger.dev.storj.io/trace/4405ea8eb44f1562
FAIL (2025-12-29 14:12:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 14:11:36.084557566 (574h42m35.955640601s) [upload=503828]: Starting upload
Dec 29 14:11:36.084631788 (574h42m35.955714823s) [upload=503828]: Done starting upload
Dec 29 14:11:36.084729925 (574h42m35.95581297s) [upload=503828]: Uploading segments...
Dec 29 14:11:36.084903916 (574h42m35.955986981s) [upload=503828]: Got next segment. Inline: false
Dec 29 14:11:36.084945695 (574h42m35.95602875s) [upload=503828]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 14:11:36.084952578 (574h42m35.956035633s) [upload=503828]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 14:11:36.332502971 (574h42m36.203586006s) [seg_pos={0 0}, upload=503828]: Begin upload segment...
Dec 29 14:11:36.332546052 (574h42m36.203629087s) [seg_pos={0 0}, upload=503828]: Done begin upload segment.
Dec 29 14:11:36.332571511 (574h42m36.203654546s) [noise=true , address=145.79.227.20:4437 , piece_id=7SR3ZDXNXQHTVHM7USP5Z7XIM4E2VNLCXIJFW4APQXXIPUU6E4RA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.332600596 (574h42m36.203683641s) [noise=true , address=45.91.116.10:4083 , piece_id=7T3S5FJJBC76U2USTDBHJT3DQHV23CEYXAKYIPA7CAXWKNLSHHEA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.334114198 (574h42m36.205197233s) [noise=true , address=77.90.31.12:4590 , piece_id=GQAN5UC26XGJ4VUS7QRFJIEWIQZBL5U4SXRYLX2WW2U67NPPMAXQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.334710514 (574h42m36.205793559s) [noise=true , address=145.79.227.45:4231 , piece_id=O3LJQPVM3WO3T4ENSCKKSRP3JR3PIVH4AJAANV7HSSA7C6QPP5EQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.335224112 (574h42m36.206307147s) [noise=true , address=145.79.227.2:4114 , piece_id=X7HDCYG4IAYA5Y22V2IXI245YQK7NYPGIFVCHENQBDXQSMZEE4OQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.336236940 (574h42m36.207319986s) [noise=true , address=77.90.31.14:4661 , piece_id=DQX6Y3JDRJ6KDXPE4K6XD7QF4HDAFENDIOMH7ITAYERVIRCAJCZA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.336813127 (574h42m36.207896172s) [noise=true , address=45.91.116.6:4973 , piece_id=AVFQHOEWBMA6UZMVLSRU7CQTCKWOSPSXB52GHZ33LN4T2VPID4NA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.337420985 (574h42m36.20850403s) [noise=true , address=145.79.227.46:4602 , piece_id=IJLUJ5GJYDTHFGEJPX7DYPQ5LL4S4YFDAP6GYFPZOXNYSG2RYD3Q, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.338295400 (574h42m36.209378435s) [noise=true , address=77.90.31.11:4556 , piece_id=ZGOBWILEVKJJG2H7XZNFNDBN5OO6AYDGSNZ5NEUSMUYJNXWFIBJQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.338760435 (574h42m36.20984349s) [noise=true , address=145.79.227.5:4223 , piece_id=4N3P6SMV44G74MLYTLMT3UBWU65CN5MIZXTFDA6WCGNKBURG5DKA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.339563494 (574h42m36.210646539s) [noise=true , address=77.90.31.15:4700 , piece_id=3AKU4HAI6ACJUV6FBQY2Y2K36CGMJAYY5BETIAX3L7HQJDUB3GOQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.340251805 (574h42m36.21133484s) [noise=true , address=145.79.227.12:4765 , piece_id=DOXR7MVGLDD6WWQ26CXTP4ZLSZAYABLCWIIZFSFLK5VC7MO5VOGA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.343805021 (574h42m36.214888056s) [noise=true , address=145.79.227.19:4077 , piece_id=OAGG6IDCOHMZR4OJOT3EROKT2WEKBB5UQGP5GGLJNQLMETDU7XOA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.345227460 (574h42m36.216310495s) [noise=true , address=45.91.116.5:4299 , piece_id=BBQ5F5KYUUXWWV2GVWZ5DRSHP7DODI5IUJS3WL7NKNAY2Q4WPYIQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.346633327 (574h42m36.217716362s) [noise=true , address=145.79.227.14:4838 , piece_id=EEQDOUE2NRVSLN4YLKKJYBJH4YZ4L5SLUIIRNKP5FUTMPWIL3ALQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.347802193 (574h42m36.218885228s) [noise=true , address=45.91.116.13:4154 , piece_id=QPPY4N7EJZ3JL37ZKQN73BEISCHU3ZDD4H53J5AU2TER62CO25TQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:38.045870619 (574h42m37.916953654s) [upload=503828]: Next returned nil segment
Dec 29 14:11:38.045885247 (574h42m37.916968282s) [upload=503828]: Waiting for error group managing segments...
Dec 29 14:11:38.328286552 (574h42m38.199369587s) [noise=true , address=145.79.227.2:4114 , piece_id=X7HDCYG4IAYA5Y22V2IXI245YQK7NYPGIFVCHENQBDXQSMZEE4OQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.329718619 (574h42m38.200801654s) [noise=true , address=145.79.227.14:4838 , piece_id=EEQDOUE2NRVSLN4YLKKJYBJH4YZ4L5SLUIIRNKP5FUTMPWIL3ALQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.335166832 (574h42m38.206249867s) [noise=true , address=145.79.227.20:4437 , piece_id=7SR3ZDXNXQHTVHM7USP5Z7XIM4E2VNLCXIJFW4APQXXIPUU6E4RA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.335591621 (574h42m38.206674676s) [noise=true , address=145.79.227.45:4231 , piece_id=O3LJQPVM3WO3T4ENSCKKSRP3JR3PIVH4AJAANV7HSSA7C6QPP5EQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.345936529 (574h42m38.217019564s) [noise=true , address=145.79.227.12:4765 , piece_id=DOXR7MVGLDD6WWQ26CXTP4ZLSZAYABLCWIIZFSFLK5VC7MO5VOGA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.346711565 (574h42m38.21779461s) [noise=true , address=145.79.227.19:4077 , piece_id=OAGG6IDCOHMZR4OJOT3EROKT2WEKBB5UQGP5GGLJNQLMETDU7XOA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.349315624 (574h42m38.220398659s) [noise=true , address=145.79.227.5:4223 , piece_id=4N3P6SMV44G74MLYTLMT3UBWU65CN5MIZXTFDA6WCGNKBURG5DKA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.352574720 (574h42m38.223657755s) [noise=true , address=145.79.227.46:4602 , piece_id=IJLUJ5GJYDTHFGEJPX7DYPQ5LL4S4YFDAP6GYFPZOXNYSG2RYD3Q, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.607098941 (574h42m38.478181976s) [noise=true , address=45.91.116.10:4083 , piece_id=7T3S5FJJBC76U2USTDBHJT3DQHV23CEYXAKYIPA7CAXWKNLSHHEA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.620967930 (574h42m38.492050975s) [noise=true , address=45.91.116.5:4299 , piece_id=BBQ5F5KYUUXWWV2GVWZ5DRSHP7DODI5IUJS3WL7NKNAY2Q4WPYIQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.682147294 (574h42m38.55323034s) [noise=true , address=77.90.31.12:4590 , piece_id=GQAN5UC26XGJ4VUS7QRFJIEWIQZBL5U4SXRYLX2WW2U67NPPMAXQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.793699226 (574h42m38.664782271s) [noise=true , address=45.91.116.6:4973 , piece_id=AVFQHOEWBMA6UZMVLSRU7CQTCKWOSPSXB52GHZ33LN4T2VPID4NA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:39.056175480 (574h42m38.927258525s) [noise=true , address=45.91.116.13:4154 , piece_id=QPPY4N7EJZ3JL37ZKQN73BEISCHU3ZDD4H53J5AU2TER62CO25TQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:39.056228642 (574h42m38.927311677s) [seg_pos={0 0}, upload=503828]: Segment reached optimal threshold of 13 pieces.
Dec 29 14:11:39.056437299 (574h42m38.927520344s) [noise=true , address=77.90.31.11:4556 , piece_id=ZGOBWILEVKJJG2H7XZNFNDBN5OO6AYDGSNZ5NEUSMUYJNXWFIBJQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 29 14:11:39.056443190 (574h42m38.927526225s) [noise=true , address=77.90.31.11:4556 , piece_id=ZGOBWILEVKJJG2H7XZNFNDBN5OO6AYDGSNZ5NEUSMUYJNXWFIBJQ, seg_pos={0 0}, upload=503828]: Long tail cancellation due to optimal threshold
Dec 29 14:11:39.056669110 (574h42m38.927752145s) [noise=true , address=77.90.31.15:4700 , piece_id=3AKU4HAI6ACJUV6FBQY2Y2K36CGMJAYY5BETIAX3L7HQJDUB3GOQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43120->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43120->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43120->77.90.31.15:4700: use of closed network connection
Dec 29 14:11:39.056677917 (574h42m38.927760952s) [noise=true , address=77.90.31.15:4700 , piece_id=3AKU4HAI6ACJUV6FBQY2Y2K36CGMJAYY5BETIAX3L7HQJDUB3GOQ, seg_pos={0 0}, upload=503828]: Long tail cancellation due to optimal threshold
Dec 29 14:11:39.056816029 (574h42m38.927899095s) [noise=true , address=77.90.31.14:4661 , piece_id=DQX6Y3JDRJ6KDXPE4K6XD7QF4HDAFENDIOMH7ITAYERVIRCAJCZA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 29 14:11:39.056827863 (574h42m38.927910908s) [noise=true , address=77.90.31.14:4661 , piece_id=DQX6Y3JDRJ6KDXPE4K6XD7QF4HDAFENDIOMH7ITAYERVIRCAJCZA, seg_pos={0 0}, upload=503828]: Long tail cancellation due to optimal threshold
Dec 29 14:11:39.056846357 (574h42m38.927929392s) [seg_pos={0 0}, upload=503828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 14:11:39.056914707 (574h42m38.927997743s) [upload=503828]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:11:39.056918865 (574h42m38.9280019s) [upload=503828]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:11:39.358295061 (574h42m39.229378096s) [upload=503828]: Done uploading segments...
Dec 29 14:11:39.358321060 (574h42m39.229404095s) [upload=503828]: Upload finished. err: <nil>
Duration: 7.079270432s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXXX] duration:
5.001616874s trace:
https://jaeger.dev.storj.io/trace/4656b818eb04f2f0
FAIL (2025-12-29 14:12:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001637893s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [.................X...........X...X....X....X.XX...........X.] duration:
39.653493354s trace:
https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd
FAIL (2025-12-29 14:12:11 UTC)
orbiter warning: context deadline exceeded
Duration: 39.653505748s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
35 out of 60 is failed (threshold=3) [.X...X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XXX] duration:
30.003443844s trace:
https://jaeger.dev.storj.io/trace/4405ea8eb44f1562
FAIL (2025-12-29 14:12:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 14:11:36.084557566 (574h42m35.955640601s) [upload=503828]: Starting upload
Dec 29 14:11:36.084631788 (574h42m35.955714823s) [upload=503828]: Done starting upload
Dec 29 14:11:36.084729925 (574h42m35.95581297s) [upload=503828]: Uploading segments...
Dec 29 14:11:36.084903916 (574h42m35.955986981s) [upload=503828]: Got next segment. Inline: false
Dec 29 14:11:36.084945695 (574h42m35.95602875s) [upload=503828]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 14:11:36.084952578 (574h42m35.956035633s) [upload=503828]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 14:11:36.332502971 (574h42m36.203586006s) [seg_pos={0 0}, upload=503828]: Begin upload segment...
Dec 29 14:11:36.332546052 (574h42m36.203629087s) [seg_pos={0 0}, upload=503828]: Done begin upload segment.
Dec 29 14:11:36.332571511 (574h42m36.203654546s) [noise=true , address=145.79.227.20:4437 , piece_id=7SR3ZDXNXQHTVHM7USP5Z7XIM4E2VNLCXIJFW4APQXXIPUU6E4RA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.332600596 (574h42m36.203683641s) [noise=true , address=45.91.116.10:4083 , piece_id=7T3S5FJJBC76U2USTDBHJT3DQHV23CEYXAKYIPA7CAXWKNLSHHEA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.334114198 (574h42m36.205197233s) [noise=true , address=77.90.31.12:4590 , piece_id=GQAN5UC26XGJ4VUS7QRFJIEWIQZBL5U4SXRYLX2WW2U67NPPMAXQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.334710514 (574h42m36.205793559s) [noise=true , address=145.79.227.45:4231 , piece_id=O3LJQPVM3WO3T4ENSCKKSRP3JR3PIVH4AJAANV7HSSA7C6QPP5EQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.335224112 (574h42m36.206307147s) [noise=true , address=145.79.227.2:4114 , piece_id=X7HDCYG4IAYA5Y22V2IXI245YQK7NYPGIFVCHENQBDXQSMZEE4OQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.336236940 (574h42m36.207319986s) [noise=true , address=77.90.31.14:4661 , piece_id=DQX6Y3JDRJ6KDXPE4K6XD7QF4HDAFENDIOMH7ITAYERVIRCAJCZA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.336813127 (574h42m36.207896172s) [noise=true , address=45.91.116.6:4973 , piece_id=AVFQHOEWBMA6UZMVLSRU7CQTCKWOSPSXB52GHZ33LN4T2VPID4NA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.337420985 (574h42m36.20850403s) [noise=true , address=145.79.227.46:4602 , piece_id=IJLUJ5GJYDTHFGEJPX7DYPQ5LL4S4YFDAP6GYFPZOXNYSG2RYD3Q, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.338295400 (574h42m36.209378435s) [noise=true , address=77.90.31.11:4556 , piece_id=ZGOBWILEVKJJG2H7XZNFNDBN5OO6AYDGSNZ5NEUSMUYJNXWFIBJQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.338760435 (574h42m36.20984349s) [noise=true , address=145.79.227.5:4223 , piece_id=4N3P6SMV44G74MLYTLMT3UBWU65CN5MIZXTFDA6WCGNKBURG5DKA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.339563494 (574h42m36.210646539s) [noise=true , address=77.90.31.15:4700 , piece_id=3AKU4HAI6ACJUV6FBQY2Y2K36CGMJAYY5BETIAX3L7HQJDUB3GOQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.340251805 (574h42m36.21133484s) [noise=true , address=145.79.227.12:4765 , piece_id=DOXR7MVGLDD6WWQ26CXTP4ZLSZAYABLCWIIZFSFLK5VC7MO5VOGA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.343805021 (574h42m36.214888056s) [noise=true , address=145.79.227.19:4077 , piece_id=OAGG6IDCOHMZR4OJOT3EROKT2WEKBB5UQGP5GGLJNQLMETDU7XOA, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.345227460 (574h42m36.216310495s) [noise=true , address=45.91.116.5:4299 , piece_id=BBQ5F5KYUUXWWV2GVWZ5DRSHP7DODI5IUJS3WL7NKNAY2Q4WPYIQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.346633327 (574h42m36.217716362s) [noise=true , address=145.79.227.14:4838 , piece_id=EEQDOUE2NRVSLN4YLKKJYBJH4YZ4L5SLUIIRNKP5FUTMPWIL3ALQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:36.347802193 (574h42m36.218885228s) [noise=true , address=45.91.116.13:4154 , piece_id=QPPY4N7EJZ3JL37ZKQN73BEISCHU3ZDD4H53J5AU2TER62CO25TQ, seg_pos={0 0}, upload=503828]: Uploading piece...
Dec 29 14:11:38.045870619 (574h42m37.916953654s) [upload=503828]: Next returned nil segment
Dec 29 14:11:38.045885247 (574h42m37.916968282s) [upload=503828]: Waiting for error group managing segments...
Dec 29 14:11:38.328286552 (574h42m38.199369587s) [noise=true , address=145.79.227.2:4114 , piece_id=X7HDCYG4IAYA5Y22V2IXI245YQK7NYPGIFVCHENQBDXQSMZEE4OQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.329718619 (574h42m38.200801654s) [noise=true , address=145.79.227.14:4838 , piece_id=EEQDOUE2NRVSLN4YLKKJYBJH4YZ4L5SLUIIRNKP5FUTMPWIL3ALQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.335166832 (574h42m38.206249867s) [noise=true , address=145.79.227.20:4437 , piece_id=7SR3ZDXNXQHTVHM7USP5Z7XIM4E2VNLCXIJFW4APQXXIPUU6E4RA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.335591621 (574h42m38.206674676s) [noise=true , address=145.79.227.45:4231 , piece_id=O3LJQPVM3WO3T4ENSCKKSRP3JR3PIVH4AJAANV7HSSA7C6QPP5EQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.345936529 (574h42m38.217019564s) [noise=true , address=145.79.227.12:4765 , piece_id=DOXR7MVGLDD6WWQ26CXTP4ZLSZAYABLCWIIZFSFLK5VC7MO5VOGA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.346711565 (574h42m38.21779461s) [noise=true , address=145.79.227.19:4077 , piece_id=OAGG6IDCOHMZR4OJOT3EROKT2WEKBB5UQGP5GGLJNQLMETDU7XOA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.349315624 (574h42m38.220398659s) [noise=true , address=145.79.227.5:4223 , piece_id=4N3P6SMV44G74MLYTLMT3UBWU65CN5MIZXTFDA6WCGNKBURG5DKA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.352574720 (574h42m38.223657755s) [noise=true , address=145.79.227.46:4602 , piece_id=IJLUJ5GJYDTHFGEJPX7DYPQ5LL4S4YFDAP6GYFPZOXNYSG2RYD3Q, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.607098941 (574h42m38.478181976s) [noise=true , address=45.91.116.10:4083 , piece_id=7T3S5FJJBC76U2USTDBHJT3DQHV23CEYXAKYIPA7CAXWKNLSHHEA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.620967930 (574h42m38.492050975s) [noise=true , address=45.91.116.5:4299 , piece_id=BBQ5F5KYUUXWWV2GVWZ5DRSHP7DODI5IUJS3WL7NKNAY2Q4WPYIQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.682147294 (574h42m38.55323034s) [noise=true , address=77.90.31.12:4590 , piece_id=GQAN5UC26XGJ4VUS7QRFJIEWIQZBL5U4SXRYLX2WW2U67NPPMAXQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:38.793699226 (574h42m38.664782271s) [noise=true , address=45.91.116.6:4973 , piece_id=AVFQHOEWBMA6UZMVLSRU7CQTCKWOSPSXB52GHZ33LN4T2VPID4NA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:39.056175480 (574h42m38.927258525s) [noise=true , address=45.91.116.13:4154 , piece_id=QPPY4N7EJZ3JL37ZKQN73BEISCHU3ZDD4H53J5AU2TER62CO25TQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: <nil>
Dec 29 14:11:39.056228642 (574h42m38.927311677s) [seg_pos={0 0}, upload=503828]: Segment reached optimal threshold of 13 pieces.
Dec 29 14:11:39.056437299 (574h42m38.927520344s) [noise=true , address=77.90.31.11:4556 , piece_id=ZGOBWILEVKJJG2H7XZNFNDBN5OO6AYDGSNZ5NEUSMUYJNXWFIBJQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 29 14:11:39.056443190 (574h42m38.927526225s) [noise=true , address=77.90.31.11:4556 , piece_id=ZGOBWILEVKJJG2H7XZNFNDBN5OO6AYDGSNZ5NEUSMUYJNXWFIBJQ, seg_pos={0 0}, upload=503828]: Long tail cancellation due to optimal threshold
Dec 29 14:11:39.056669110 (574h42m38.927752145s) [noise=true , address=77.90.31.15:4700 , piece_id=3AKU4HAI6ACJUV6FBQY2Y2K36CGMJAYY5BETIAX3L7HQJDUB3GOQ, seg_pos={0 0}, upload=503828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43120->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43120->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43120->77.90.31.15:4700: use of closed network connection
Dec 29 14:11:39.056677917 (574h42m38.927760952s) [noise=true , address=77.90.31.15:4700 , piece_id=3AKU4HAI6ACJUV6FBQY2Y2K36CGMJAYY5BETIAX3L7HQJDUB3GOQ, seg_pos={0 0}, upload=503828]: Long tail cancellation due to optimal threshold
Dec 29 14:11:39.056816029 (574h42m38.927899095s) [noise=true , address=77.90.31.14:4661 , piece_id=DQX6Y3JDRJ6KDXPE4K6XD7QF4HDAFENDIOMH7ITAYERVIRCAJCZA, seg_pos={0 0}, upload=503828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 29 14:11:39.056827863 (574h42m38.927910908s) [noise=true , address=77.90.31.14:4661 , piece_id=DQX6Y3JDRJ6KDXPE4K6XD7QF4HDAFENDIOMH7ITAYERVIRCAJCZA, seg_pos={0 0}, upload=503828]: Long tail cancellation due to optimal threshold
Dec 29 14:11:39.056846357 (574h42m38.927929392s) [seg_pos={0 0}, upload=503828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 14:11:39.056914707 (574h42m38.927997743s) [upload=503828]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:11:39.056918865 (574h42m38.9280019s) [upload=503828]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:11:39.358295061 (574h42m39.229378096s) [upload=503828]: Done uploading segments...
Dec 29 14:11:39.358321060 (574h42m39.229404095s) [upload=503828]: Upload finished. err: <nil>
Duration: 30.003467911s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [........................................XXX..X..XX..........] duration:
46.760002487s trace:
https://jaeger.dev.storj.io/trace/266137264da4c90
FAIL (2025-12-29 14:11:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded
======LOGS======
NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment
Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments...
Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection
Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection
Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection
Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments...
Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil>
Duration: 46.760021743s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.................X...X.X.XX....X........X.X............X..X.] duration:
23.768107216s trace:
https://jaeger.dev.storj.io/trace/21851d989b155b7f
FAIL (2025-12-29 14:11:45 UTC)
orbiter warning: context deadline exceeded
Duration: 23.768127033s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..................X...........X...X....X....X.XX...........X] duration:
1m0.001542004s trace:
https://jaeger.dev.storj.io/trace/1fb97dc6a9466afd
FAIL (2025-12-29 14:11:31 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.00156651s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXXX] duration:
5.001300998s trace:
https://jaeger.dev.storj.io/trace/16520eb71e3ce0c4
FAIL (2025-12-29 14:11:26 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00131196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
34 out of 60 is failed (threshold=3) [..X...X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.XX] duration:
30.004080519s trace:
https://jaeger.dev.storj.io/trace/5813c5e089e39b2f
FAIL (2025-12-29 14:11:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 4LVI7SO3QYXFAGNSES5IPBVDMZKGWMSTDZSAUKIX4YMAJQ7ZGZJQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: M66TKGBQNRJRPN4PWTETFZGLIDBHOM34ECYEQ5TCMWYAMRMUO57A) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: ZLKDJHGTKNEM55C535LFNBDFQKNQEZD4PLSONSVMOXPCTGJ3GBBQ) context deadline exceeded
======LOGS======
Dec 29 14:10:51.937878798 (574h41m51.808961843s) [upload=503816]: Starting upload
Dec 29 14:10:51.938172067 (574h41m51.809255122s) [upload=503816]: Done starting upload
Dec 29 14:10:51.938277288 (574h41m51.809360333s) [upload=503816]: Uploading segments...
Dec 29 14:10:51.938485995 (574h41m51.80956903s) [upload=503816]: Got next segment. Inline: false
Dec 29 14:10:51.938547362 (574h41m51.809630407s) [upload=503816]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 14:10:51.938554175 (574h41m51.80963722s) [upload=503816]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 14:10:52.186172947 (574h41m52.057255982s) [seg_pos={0 0}, upload=503816]: Begin upload segment...
Dec 29 14:10:52.186228273 (574h41m52.057311308s) [seg_pos={0 0}, upload=503816]: Done begin upload segment.
Dec 29 14:10:52.186253211 (574h41m52.057336246s) [noise=true , address=45.91.116.8:4045 , piece_id=M66TKGBQNRJRPN4PWTETFZGLIDBHOM34ECYEQ5TCMWYAMRMUO57A, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.186359042 (574h41m52.057442078s) [noise=true , address=45.91.116.9:4261 , piece_id=FZOWUDPKCCW7UOH5ZHOR6CQNAP7L5QKPLWORJE7JTSWCRVTTHYKQ, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.186917566 (574h41m52.058000611s) [noise=true , address=77.90.31.14:4663 , piece_id=ZLKDJHGTKNEM55C535LFNBDFQKNQEZD4PLSONSVMOXPCTGJ3GBBQ, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.187017216 (574h41m52.058100261s) [noise=true , address=145.79.227.7:4294 , piece_id=4YWT6FNL3OAPQEHLYPXOL4Q3R7QWL3WD2YUA2YA6JFWTEUNYU57Q, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.187538938 (574h41m52.058621973s) [noise=true , address=77.90.31.10:4519 , piece_id=W2MSFOYPMI3SG6LRVFHMEZQGAVZ3VY3BGE3T2HLQDTVTJT2VFUGA, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.187819103 (574h41m52.058902148s) [noise=true , address=145.79.227.15:4876 , piece_id=3NV5GIXIMVB2NOA6XGUXA3H3GIH6335PM4W4UIBPSQZNATC4REOA, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.188068698 (574h41m52.059151733s) [noise=true , address=145.79.227.34:4905 , piece_id=MWK3CE2IG53PABEGJ5363VOUTRMBHQFXDOV2MESIJACLNRPY6N7A, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.188167266 (574h41m52.059250301s) [noise=true , address=145.79.227.10:4401 , piece_id=3Z6TECTU44OURT5JG5ANLWK7LNAJQQLCE2ARP536V2QSDDWNMOYQ, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.188312973 (574h41m52.059396028s) [noise=true , address=45.91.116.10:4083 , piece_id=RBT4OOUWW2ELPVL6IDRRRTSLDGICDK5KFHH7LN2PPESD4NBC4YYQ, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.188603005 (574h41m52.05968604s) [noise=true , address=45.91.116.5:4299 , piece_id=4LVI7SO3QYXFAGNSES5IPBVDMZKGWMSTDZSAUKIX4YMAJQ7ZGZJQ, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.188695351 (574h41m52.059778396s) [noise=true , address=145.79.227.49:4337 , piece_id=H3D52UN3KW6PL5P5MGWHJU3YFERF3ODRWZMJ3IIHLJWZIRYXPO2Q, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.188872248 (574h41m52.059955283s) [noise=true , address=145.79.227.37:4565 , piece_id=LUNLB2MIFUGQLLDAMEUGB4JNEZSCVDAOGJTCUD6MBCYKFA3UDLMQ, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.189071177 (574h41m52.060154212s) [noise=true , address=145.79.227.52:4445 , piece_id=VINXBSZJURF7OVZE2KO2UJ7ZQILVGAJ4I5FZTHSPKTFUSABQSQEQ, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.189416053 (574h41m52.060499098s) [noise=true , address=45.91.116.11:4120 , piece_id=WKTZWIDZG5WQLELRP7XKKVMW77JQCSVS2U3EX5B6V3OYGGFYVILA, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.189731313 (574h41m52.060814358s) [noise=true , address=45.91.116.4:4940 , piece_id=AAF5MOOZCZUZZCILRM3ZTWJ2UQ4WD2XB2C7BBZ65J6S6DNSZ525A, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:52.190296039 (574h41m52.061379074s) [noise=true , address=145.79.227.48:4301 , piece_id=QK2W5AHJHKOMAFXZ2EZ2THWTPATH2BAHXL3625DTO45DEXVHC2CA, seg_pos={0 0}, upload=503816]: Uploading piece...
Dec 29 14:10:53.878252558 (574h41m53.749335593s) [upload=503816]: Next returned nil segment
Dec 29 14:10:53.878321379 (574h41m53.749404454s) [upload=503816]: Waiting for error group managing segments...
Dec 29 14:10:54.171610916 (574h41m54.042693971s) [noise=true , address=145.79.227.10:4401 , piece_id=3Z6TECTU44OURT5JG5ANLWK7LNAJQQLCE2ARP536V2QSDDWNMOYQ, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.172032489 (574h41m54.043115534s) [noise=true , address=145.79.227.15:4876 , piece_id=3NV5GIXIMVB2NOA6XGUXA3H3GIH6335PM4W4UIBPSQZNATC4REOA, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.178582230 (574h41m54.049665275s) [noise=true , address=145.79.227.37:4565 , piece_id=LUNLB2MIFUGQLLDAMEUGB4JNEZSCVDAOGJTCUD6MBCYKFA3UDLMQ, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.183430742 (574h41m54.054513787s) [noise=true , address=145.79.227.48:4301 , piece_id=QK2W5AHJHKOMAFXZ2EZ2THWTPATH2BAHXL3625DTO45DEXVHC2CA, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.187709540 (574h41m54.058792575s) [noise=true , address=145.79.227.52:4445 , piece_id=VINXBSZJURF7OVZE2KO2UJ7ZQILVGAJ4I5FZTHSPKTFUSABQSQEQ, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.202869788 (574h41m54.073952833s) [noise=true , address=145.79.227.7:4294 , piece_id=4YWT6FNL3OAPQEHLYPXOL4Q3R7QWL3WD2YUA2YA6JFWTEUNYU57Q, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.426086365 (574h41m54.29716941s) [noise=true , address=45.91.116.11:4120 , piece_id=WKTZWIDZG5WQLELRP7XKKVMW77JQCSVS2U3EX5B6V3OYGGFYVILA, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.439949082 (574h41m54.311032127s) [noise=true , address=45.91.116.4:4940 , piece_id=AAF5MOOZCZUZZCILRM3ZTWJ2UQ4WD2XB2C7BBZ65J6S6DNSZ525A, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.441406727 (574h41m54.312489772s) [noise=true , address=45.91.116.8:4045 , piece_id=M66TKGBQNRJRPN4PWTETFZGLIDBHOM34ECYEQ5TCMWYAMRMUO57A, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.506688734 (574h41m54.377771779s) [noise=true , address=77.90.31.10:4519 , piece_id=W2MSFOYPMI3SG6LRVFHMEZQGAVZ3VY3BGE3T2HLQDTVTJT2VFUGA, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.559472552 (574h41m54.430555588s) [noise=true , address=45.91.116.5:4299 , piece_id=4LVI7SO3QYXFAGNSES5IPBVDMZKGWMSTDZSAUKIX4YMAJQ7ZGZJQ, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.561780908 (574h41m54.432863953s) [noise=true , address=45.91.116.9:4261 , piece_id=FZOWUDPKCCW7UOH5ZHOR6CQNAP7L5QKPLWORJE7JTSWCRVTTHYKQ, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.574999949 (574h41m54.446082994s) [noise=true , address=77.90.31.14:4663 , piece_id=ZLKDJHGTKNEM55C535LFNBDFQKNQEZD4PLSONSVMOXPCTGJ3GBBQ, seg_pos={0 0}, upload=503816]: Done uploading piece. err: <nil>
Dec 29 14:10:54.575026980 (574h41m54.446110025s) [seg_pos={0 0}, upload=503816]: Segment reached optimal threshold of 13 pieces.
Dec 29 14:10:54.575333183 (574h41m54.446416228s) [noise=true , address=145.79.227.49:4337 , piece_id=H3D52UN3KW6PL5P5MGWHJU3YFERF3ODRWZMJ3IIHLJWZIRYXPO2Q, seg_pos={0 0}, upload=503816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 29 14:10:54.575343503 (574h41m54.446426548s) [noise=true , address=145.79.227.49:4337 , piece_id=H3D52UN3KW6PL5P5MGWHJU3YFERF3ODRWZMJ3IIHLJWZIRYXPO2Q, seg_pos={0 0}, upload=503816]: Long tail cancellation due to optimal threshold
Dec 29 14:10:54.575421893 (574h41m54.446504928s) [noise=true , address=145.79.227.34:4905 , piece_id=MWK3CE2IG53PABEGJ5363VOUTRMBHQFXDOV2MESIJACLNRPY6N7A, seg_pos={0 0}, upload=503816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 29 14:10:54.575439967 (574h41m54.446523012s) [noise=true , address=145.79.227.34:4905 , piece_id=MWK3CE2IG53PABEGJ5363VOUTRMBHQFXDOV2MESIJACLNRPY6N7A, seg_pos={0 0}, upload=503816]: Long tail cancellation due to optimal threshold
Dec 29 14:10:54.575641891 (574h41m54.446724946s) [noise=true , address=45.91.116.10:4083 , piece_id=RBT4OOUWW2ELPVL6IDRRRTSLDGICDK5KFHH7LN2PPESD4NBC4YYQ, seg_pos={0 0}, upload=503816]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 29 14:10:54.575652090 (574h41m54.446735135s) [noise=true , address=45.91.116.10:4083 , piece_id=RBT4OOUWW2ELPVL6IDRRRTSLDGICDK5KFHH7LN2PPESD4NBC4YYQ, seg_pos={0 0}, upload=503816]: Long tail cancellation due to optimal threshold
Dec 29 14:10:54.575680805 (574h41m54.44676385s) [seg_pos={0 0}, upload=503816]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 14:10:54.575771528 (574h41m54.446854573s) [upload=503816]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:10:54.575779894 (574h41m54.446862939s) [upload=503816]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:10:54.866376383 (574h41m54.737459428s) [upload=503816]: Done uploading segments...
Dec 29 14:10:54.866437059 (574h41m54.737520114s) [upload=503816]: Upload finished. err: <nil>
Duration: 30.004114243s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.........................................XXX..X..XX.........] duration:
48.053913462s trace:
https://jaeger.dev.storj.io/trace/266137264da4c90
FAIL (2025-12-29 14:11:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded
======LOGS======
NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment
Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments...
Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection
Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection
Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection
Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments...
Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil>
Duration: 48.053941366s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXXX] duration:
5.001123425s trace:
https://jaeger.dev.storj.io/trace/1013eb59c5099f9f
FAIL (2025-12-29 14:10:48 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001145587s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [..................X...X.X.XX....X........X.X............X..X] duration:
30.001753747s trace:
https://jaeger.dev.storj.io/trace/21851d989b155b7f
FAIL (2025-12-29 14:10:46 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001774767s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
34 out of 60 is failed (threshold=3) [X..X...X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.X] duration:
30.002611887s trace:
https://jaeger.dev.storj.io/trace/7037737383e22050
FAIL (2025-12-29 14:10:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: VIESTVG6IKJCBKCHFQCNKI7WKCO27BNARPNTNOITUD3R6EKK46MQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 6S7HNYSQPCNT3XFL6X3ERZO7YGPM6DQB72CSH7B5ZQE4ILNQ6YNA) context deadline exceeded
======LOGS======
Dec 29 14:09:55.066786337 (574h40m54.937869382s) [upload=503804]: Starting upload
Dec 29 14:09:55.066866480 (574h40m54.937949525s) [upload=503804]: Done starting upload
Dec 29 14:09:55.067132876 (574h40m54.938215921s) [upload=503804]: Uploading segments...
Dec 29 14:09:55.067173765 (574h40m54.93825681s) [upload=503804]: Got next segment. Inline: false
Dec 29 14:09:55.067203652 (574h40m54.938286697s) [upload=503804]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 14:09:55.067209513 (574h40m54.938292578s) [upload=503804]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 14:09:55.313041434 (574h40m55.184124479s) [seg_pos={0 0}, upload=503804]: Begin upload segment...
Dec 29 14:09:55.313105385 (574h40m55.18418843s) [seg_pos={0 0}, upload=503804]: Done begin upload segment.
Dec 29 14:09:55.313134852 (574h40m55.184217897s) [noise=true , address=45.91.116.13:4153 , piece_id=C3NYI65X3YFRIE3GAU5OCNJOME2ZTXPFSMUOLJVEBZNS4EJWEOWQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.313144310 (574h40m55.184227345s) [noise=true , address=145.79.227.45:4230 , piece_id=W3YXZLXKRZ5ZWG43MMOSONYSP32YHS3VQQVVS6O3SUSAPKDQF6VQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.313902193 (574h40m55.184985238s) [noise=true , address=145.79.227.4:4187 , piece_id=3ULQX7WEMYMRCANSNTBYZRONPYEKDLXB4YVSMQGOL3I6ZCGZISLA, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.317008477 (574h40m55.188091512s) [noise=true , address=145.79.227.12:4767 , piece_id=76RQJG2XMCRIRNJU7SYUN6IOQSPAE7DCE3YECFL5F3LXASPRWFSQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.318910600 (574h40m55.189993635s) [noise=true , address=45.91.116.14:4192 , piece_id=VIESTVG6IKJCBKCHFQCNKI7WKCO27BNARPNTNOITUD3R6EKK46MQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.320526206 (574h40m55.191609251s) [noise=true , address=145.79.227.14:4838 , piece_id=J2UNRCFUUWL3C4ICFIXDRNTGOX3JNFCFFLLUMQ3344DR7MQ7KH3A, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.321553082 (574h40m55.192636127s) [noise=true , address=145.79.227.52:4446 , piece_id=7CC7R72KLMIRLG45E4RGQMH55X7EDKSFXXQLA7IKGQ6IABBND7RQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.322271409 (574h40m55.193354464s) [noise=true , address=145.79.227.11:4730 , piece_id=WO4VKT6ZQ24YDII2NRCYKVOJZZRQVJSQB3YKVNHCLGAZ6U6SGNOQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.323831920 (574h40m55.194914965s) [noise=true , address=77.90.31.19:4841 , piece_id=2VJYO6A6HQ4ZS2KLC5MSWHA7RU3PLYVWECLZ3D3KNQ6OSI3MHPRA, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.323922953 (574h40m55.195005998s) [noise=true , address=45.91.116.11:4120 , piece_id=6S7HNYSQPCNT3XFL6X3ERZO7YGPM6DQB72CSH7B5ZQE4ILNQ6YNA, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.325611940 (574h40m55.196694985s) [noise=true , address=45.91.116.12:4226 , piece_id=2PCIBCMVXJJBXG4XVYF2KTKANAFIW7WBLXR5DNQUFMGHMZIKAY7A, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.327103149 (574h40m55.198186204s) [noise=true , address=77.90.31.17:4773 , piece_id=DDGNWAJRDKYNQNTCFQZGXAHYVF2EJQIACZEKVGULBECQLDAGC5VQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.328084668 (574h40m55.199167703s) [noise=true , address=145.79.227.13:4804 , piece_id=WUERDDUDMLAILQZKBW7CCFUVGGQFIIHHEN2SV4CASZNDPYXE7ZSQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.329921495 (574h40m55.20100454s) [noise=true , address=145.79.227.7:4295 , piece_id=OT7K3IXWIWVXCANHIJ4LIGUMAZGK4HG2GRU3Q4QU3M7B2EKVIKAQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.331024335 (574h40m55.20210738s) [noise=true , address=145.79.227.36:4519 , piece_id=G45F3O3OG4PYU6M6TNQOQ7E5PRD2UAHUU5C37GTI3BAXZAOES2DQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:55.332992283 (574h40m55.204075328s) [noise=true , address=45.91.116.4:4940 , piece_id=ZATOCGHFT53GLE4PM5BEVNPYTRSWSTSVDXW4UZDTY2T4OD3I5LNQ, seg_pos={0 0}, upload=503804]: Uploading piece...
Dec 29 14:09:57.023906155 (574h40m56.89498919s) [upload=503804]: Next returned nil segment
Dec 29 14:09:57.023921053 (574h40m56.895004088s) [upload=503804]: Waiting for error group managing segments...
Dec 29 14:09:57.296859779 (574h40m57.167942814s) [noise=true , address=145.79.227.11:4730 , piece_id=WO4VKT6ZQ24YDII2NRCYKVOJZZRQVJSQB3YKVNHCLGAZ6U6SGNOQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.304647207 (574h40m57.175730242s) [noise=true , address=145.79.227.14:4838 , piece_id=J2UNRCFUUWL3C4ICFIXDRNTGOX3JNFCFFLLUMQ3344DR7MQ7KH3A, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.315454195 (574h40m57.18653724s) [noise=true , address=145.79.227.52:4446 , piece_id=7CC7R72KLMIRLG45E4RGQMH55X7EDKSFXXQLA7IKGQ6IABBND7RQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.320482349 (574h40m57.191565384s) [noise=true , address=145.79.227.36:4519 , piece_id=G45F3O3OG4PYU6M6TNQOQ7E5PRD2UAHUU5C37GTI3BAXZAOES2DQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.320623748 (574h40m57.191706794s) [noise=true , address=145.79.227.7:4295 , piece_id=OT7K3IXWIWVXCANHIJ4LIGUMAZGK4HG2GRU3Q4QU3M7B2EKVIKAQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.323812721 (574h40m57.194895756s) [noise=true , address=145.79.227.4:4187 , piece_id=3ULQX7WEMYMRCANSNTBYZRONPYEKDLXB4YVSMQGOL3I6ZCGZISLA, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.328526056 (574h40m57.199609091s) [noise=true , address=145.79.227.12:4767 , piece_id=76RQJG2XMCRIRNJU7SYUN6IOQSPAE7DCE3YECFL5F3LXASPRWFSQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.333335013 (574h40m57.204418048s) [noise=true , address=145.79.227.13:4804 , piece_id=WUERDDUDMLAILQZKBW7CCFUVGGQFIIHHEN2SV4CASZNDPYXE7ZSQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.333486652 (574h40m57.204569697s) [noise=true , address=145.79.227.45:4230 , piece_id=W3YXZLXKRZ5ZWG43MMOSONYSP32YHS3VQQVVS6O3SUSAPKDQF6VQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.598365952 (574h40m57.469448987s) [noise=true , address=77.90.31.19:4841 , piece_id=2VJYO6A6HQ4ZS2KLC5MSWHA7RU3PLYVWECLZ3D3KNQ6OSI3MHPRA, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.598890691 (574h40m57.469973736s) [noise=true , address=45.91.116.14:4192 , piece_id=VIESTVG6IKJCBKCHFQCNKI7WKCO27BNARPNTNOITUD3R6EKK46MQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.693901666 (574h40m57.564984701s) [noise=true , address=45.91.116.13:4153 , piece_id=C3NYI65X3YFRIE3GAU5OCNJOME2ZTXPFSMUOLJVEBZNS4EJWEOWQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.699824955 (574h40m57.570908s) [noise=true , address=45.91.116.11:4120 , piece_id=6S7HNYSQPCNT3XFL6X3ERZO7YGPM6DQB72CSH7B5ZQE4ILNQ6YNA, seg_pos={0 0}, upload=503804]: Done uploading piece. err: <nil>
Dec 29 14:09:57.699862526 (574h40m57.570945571s) [seg_pos={0 0}, upload=503804]: Segment reached optimal threshold of 13 pieces.
Dec 29 14:09:57.700135707 (574h40m57.571218742s) [noise=true , address=45.91.116.4:4940 , piece_id=ZATOCGHFT53GLE4PM5BEVNPYTRSWSTSVDXW4UZDTY2T4OD3I5LNQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:44384->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:44384->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44384->45.91.116.4:4940: use of closed network connection
Dec 29 14:09:57.700148671 (574h40m57.571231716s) [noise=true , address=45.91.116.4:4940 , piece_id=ZATOCGHFT53GLE4PM5BEVNPYTRSWSTSVDXW4UZDTY2T4OD3I5LNQ, seg_pos={0 0}, upload=503804]: Long tail cancellation due to optimal threshold
Dec 29 14:09:57.700221159 (574h40m57.571304224s) [noise=true , address=45.91.116.12:4226 , piece_id=2PCIBCMVXJJBXG4XVYF2KTKANAFIW7WBLXR5DNQUFMGHMZIKAY7A, seg_pos={0 0}, upload=503804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 29 14:09:57.700231067 (574h40m57.571314123s) [noise=true , address=45.91.116.12:4226 , piece_id=2PCIBCMVXJJBXG4XVYF2KTKANAFIW7WBLXR5DNQUFMGHMZIKAY7A, seg_pos={0 0}, upload=503804]: Long tail cancellation due to optimal threshold
Dec 29 14:09:57.700383208 (574h40m57.571466243s) [noise=true , address=77.90.31.17:4773 , piece_id=DDGNWAJRDKYNQNTCFQZGXAHYVF2EJQIACZEKVGULBECQLDAGC5VQ, seg_pos={0 0}, upload=503804]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 29 14:09:57.700389940 (574h40m57.571472985s) [noise=true , address=77.90.31.17:4773 , piece_id=DDGNWAJRDKYNQNTCFQZGXAHYVF2EJQIACZEKVGULBECQLDAGC5VQ, seg_pos={0 0}, upload=503804]: Long tail cancellation due to optimal threshold
Dec 29 14:09:57.700405129 (574h40m57.571488164s) [seg_pos={0 0}, upload=503804]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 14:09:57.700507434 (574h40m57.571590469s) [upload=503804]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:09:57.700521010 (574h40m57.571604045s) [upload=503804]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:09:57.987307475 (574h40m57.85839052s) [upload=503804]: Done uploading segments...
Dec 29 14:09:57.987323865 (574h40m57.858406901s) [upload=503804]: Upload finished. err: <nil>
Duration: 30.002637355s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [...................X...........X...X....X....X.XX...........] duration:
15.21884199s trace:
https://jaeger.dev.storj.io/trace/5dd0c6b12327bd0a
FAIL (2025-12-29 14:10:19 UTC)
orbiter warning: context deadline exceeded
Duration: 15.218871456s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..........................................XXX..X..XX........] duration:
33.276517421s trace:
https://jaeger.dev.storj.io/trace/266137264da4c90
FAIL (2025-12-29 14:10:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded
======LOGS======
NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment
Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments...
Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection
Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection
Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection
Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments...
Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil>
Duration: 33.276533111s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXXX] duration:
5.001516783s trace:
https://jaeger.dev.storj.io/trace/1cef3af26ccaba39
FAIL (2025-12-29 14:10:06 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001542213s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...................X...X.X.XX....X........X.X............X..] duration:
17.098053167s trace:
https://jaeger.dev.storj.io/trace/6c01aa8d1f60b948
FAIL (2025-12-29 14:09:52 UTC)
orbiter warning: context deadline exceeded
Duration: 17.098068887s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...........................................XXX..X..XX.......] duration:
53.201631509s trace:
https://jaeger.dev.storj.io/trace/266137264da4c90
FAIL (2025-12-29 14:09:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded
======LOGS======
NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment
Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments...
Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection
Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection
Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection
Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments...
Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil>
Duration: 53.201646609s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [....................X...X.X.XX....X........X.X............X.] duration:
30.002551937s trace:
https://jaeger.dev.storj.io/trace/6c01aa8d1f60b948
FAIL (2025-12-29 14:09:34 UTC)
orbiter warning: context deadline exceeded
Duration: 30.002577886s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [.X..X...X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX.] duration:
6.309344822s trace:
https://jaeger.dev.storj.io/trace/d95840e5160db39
FAIL (2025-12-29 14:09:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: GR5O4ZCGOG4ZJCX6LUDBYNWY6QV7G5V4EFTR77E7ZXFFHAXVSA2Q) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: BUI3VAXETDPZLSKE6WWEB2RZHYCTURE6PJGUAWIL4QRSPVJBDWSQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: UXK5HV77MFD2VCPT4UJ2CHANYITS7MAHRLSZUTPTOV5O4ZFEOIAQ) context deadline exceeded
======LOGS======
segments...
Dec 29 14:08:54.984379378 (574h39m54.855462413s) [upload=503789]: Got next segment. Inline: false
Dec 29 14:08:54.984422079 (574h39m54.855505114s) [upload=503789]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 14:08:54.984428572 (574h39m54.855511607s) [upload=503789]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 14:08:55.232149008 (574h39m55.103232053s) [seg_pos={0 0}, upload=503789]: Begin upload segment...
Dec 29 14:08:55.232187962 (574h39m55.103271007s) [seg_pos={0 0}, upload=503789]: Done begin upload segment.
Dec 29 14:08:55.232216888 (574h39m55.103299943s) [noise=true , address=145.79.227.36:4519 , piece_id=FAM6EJKRUJKH45X5XA4IKSAGDSXJF2M2U2HWKVU7WSCKYINYRU6A, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.232253026 (574h39m55.103336071s) [noise=true , address=45.91.116.5:4299 , piece_id=UXK5HV77MFD2VCPT4UJ2CHANYITS7MAHRLSZUTPTOV5O4ZFEOIAQ, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.232620637 (574h39m55.103703672s) [noise=true , address=145.79.227.16:4911 , piece_id=6CPTKA4R6Z7NOCET76CFGO43MRQNBXAC5GEI5CAQSS4XN4DDQ5UA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.232629594 (574h39m55.103712629s) [noise=true , address=145.79.227.15:4875 , piece_id=ES2BZXFNLVSXZ3GJB6V7HL5O3L5ADI6OUW6XERBHF5PAX3AXHEWQ, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.232933292 (574h39m55.104016337s) [noise=true , address=77.90.31.14:4661 , piece_id=GR5O4ZCGOG4ZJCX6LUDBYNWY6QV7G5V4EFTR77E7ZXFFHAXVSA2Q, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233056236 (574h39m55.104139271s) [noise=true , address=77.90.31.21:4880 , piece_id=FDD6SMQQBEWH4MUKXTKXGWSTP6IDPEWRVZRDSTKLYZTPPNSN4W4Q, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233255676 (574h39m55.104338721s) [noise=true , address=45.91.116.4:4937 , piece_id=KM72SWE6TK7LLBMWBZTMGHA7VZYCPUEL6EE5AJYQSHTZXNB736OQ, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233345988 (574h39m55.104429033s) [noise=true , address=145.79.227.14:4839 , piece_id=BBKOWRIZJBDOCJKUY4WZR5SVS4AHLOGWNFFT33AE66NQCLLAUPEA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233538174 (574h39m55.104621209s) [noise=true , address=45.91.116.10:4084 , piece_id=WZGTISH7U57JMU27FPZDOR7JECEKX6BQ3P7SOXSQVNICDU2VAOUA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233719138 (574h39m55.104802173s) [noise=true , address=145.79.227.42:4123 , piece_id=4WKBVAFEMMRUG7ACQOAK2WCCJCRJVFZ52FCQR2AE7QA6YFWY7IVA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233878271 (574h39m55.104961316s) [noise=true , address=45.91.116.13:4156 , piece_id=BUI3VAXETDPZLSKE6WWEB2RZHYCTURE6PJGUAWIL4QRSPVJBDWSQ, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.234172571 (574h39m55.105255606s) [noise=true , address=145.79.227.47:4265 , piece_id=G3VSWQDAVF3IUKVAYSZM4SOXZ7HWM2YV4CFMTNZKDPCWRZXZH5EA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.234295366 (574h39m55.105378411s) [noise=true , address=77.90.31.16:4735 , piece_id=RHAAEZJQ2DMYRLKC5BHWVNJ3ABKIAYVLYS3UJI5PH357F6QMNOPA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.234669588 (574h39m55.105752624s) [noise=true , address=145.79.227.39:4014 , piece_id=DKGPCCIHJWASXT6BOAYVJNKYKQDHKIEGQEXUJZLOSNRCI2ZFLHHA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.234740844 (574h39m55.105823889s) [noise=true , address=77.90.31.15:4697 , piece_id=ILUM2L5VOI47K7OVO6RKP3DAYXVWA6HEC7HBVHLGRJ4OLVPPDT7A, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.235331178 (574h39m55.106414223s) [noise=true , address=145.79.227.45:4231 , piece_id=CF5G3MSPJIXHEKLNOXCM4IIQAQGQORNGUNNW4NAQYUHZO4HTC2BA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:56.935848616 (574h39m56.806931661s) [upload=503789]: Next returned nil segment
Dec 29 14:08:56.935862312 (574h39m56.806945347s) [upload=503789]: Waiting for error group managing segments...
Dec 29 14:08:57.216618483 (574h39m57.087701519s) [noise=true , address=145.79.227.39:4014 , piece_id=DKGPCCIHJWASXT6BOAYVJNKYKQDHKIEGQEXUJZLOSNRCI2ZFLHHA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.222446151 (574h39m57.093529196s) [noise=true , address=145.79.227.14:4839 , piece_id=BBKOWRIZJBDOCJKUY4WZR5SVS4AHLOGWNFFT33AE66NQCLLAUPEA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.222625933 (574h39m57.093708968s) [noise=true , address=145.79.227.15:4875 , piece_id=ES2BZXFNLVSXZ3GJB6V7HL5O3L5ADI6OUW6XERBHF5PAX3AXHEWQ, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.222951603 (574h39m57.094034648s) [noise=true , address=145.79.227.16:4911 , piece_id=6CPTKA4R6Z7NOCET76CFGO43MRQNBXAC5GEI5CAQSS4XN4DDQ5UA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.223015545 (574h39m57.09409858s) [noise=true , address=145.79.227.45:4231 , piece_id=CF5G3MSPJIXHEKLNOXCM4IIQAQGQORNGUNNW4NAQYUHZO4HTC2BA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.237053956 (574h39m57.108137001s) [noise=true , address=145.79.227.47:4265 , piece_id=G3VSWQDAVF3IUKVAYSZM4SOXZ7HWM2YV4CFMTNZKDPCWRZXZH5EA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.254897743 (574h39m57.125980778s) [noise=true , address=145.79.227.42:4123 , piece_id=4WKBVAFEMMRUG7ACQOAK2WCCJCRJVFZ52FCQR2AE7QA6YFWY7IVA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.274718345 (574h39m57.14580139s) [noise=true , address=145.79.227.36:4519 , piece_id=FAM6EJKRUJKH45X5XA4IKSAGDSXJF2M2U2HWKVU7WSCKYINYRU6A, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.445595220 (574h39m57.316678255s) [noise=true , address=77.90.31.14:4661 , piece_id=GR5O4ZCGOG4ZJCX6LUDBYNWY6QV7G5V4EFTR77E7ZXFFHAXVSA2Q, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.497711177 (574h39m57.368794222s) [noise=true , address=45.91.116.13:4156 , piece_id=BUI3VAXETDPZLSKE6WWEB2RZHYCTURE6PJGUAWIL4QRSPVJBDWSQ, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.501569313 (574h39m57.372652358s) [noise=true , address=45.91.116.5:4299 , piece_id=UXK5HV77MFD2VCPT4UJ2CHANYITS7MAHRLSZUTPTOV5O4ZFEOIAQ, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.509095373 (574h39m57.380178408s) [noise=true , address=77.90.31.16:4735 , piece_id=RHAAEZJQ2DMYRLKC5BHWVNJ3ABKIAYVLYS3UJI5PH357F6QMNOPA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.624926363 (574h39m57.496009398s) [noise=true , address=45.91.116.10:4084 , piece_id=WZGTISH7U57JMU27FPZDOR7JECEKX6BQ3P7SOXSQVNICDU2VAOUA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.624950658 (574h39m57.496033703s) [seg_pos={0 0}, upload=503789]: Segment reached optimal threshold of 13 pieces.
Dec 29 14:08:57.625146321 (574h39m57.496229356s) [noise=true , address=77.90.31.15:4697 , piece_id=ILUM2L5VOI47K7OVO6RKP3DAYXVWA6HEC7HBVHLGRJ4OLVPPDT7A, seg_pos={0 0}, upload=503789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:54892->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:54892->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54892->77.90.31.15:4697: use of closed network connection
Dec 29 14:08:57.625152713 (574h39m57.496235748s) [noise=true , address=77.90.31.15:4697 , piece_id=ILUM2L5VOI47K7OVO6RKP3DAYXVWA6HEC7HBVHLGRJ4OLVPPDT7A, seg_pos={0 0}, upload=503789]: Long tail cancellation due to optimal threshold
Dec 29 14:08:57.625327216 (574h39m57.496410261s) [noise=true , address=77.90.31.21:4880 , piece_id=FDD6SMQQBEWH4MUKXTKXGWSTP6IDPEWRVZRDSTKLYZTPPNSN4W4Q, seg_pos={0 0}, upload=503789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 29 14:08:57.625343327 (574h39m57.496426362s) [noise=true , address=77.90.31.21:4880 , piece_id=FDD6SMQQBEWH4MUKXTKXGWSTP6IDPEWRVZRDSTKLYZTPPNSN4W4Q, seg_pos={0 0}, upload=503789]: Long tail cancellation due to optimal threshold
Dec 29 14:08:57.625452785 (574h39m57.49653582s) [noise=true , address=45.91.116.4:4937 , piece_id=KM72SWE6TK7LLBMWBZTMGHA7VZYCPUEL6EE5AJYQSHTZXNB736OQ, seg_pos={0 0}, upload=503789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:48452->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:48452->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48452->45.91.116.4:4937: use of closed network connection
Dec 29 14:08:57.625463064 (574h39m57.496546099s) [noise=true , address=45.91.116.4:4937 , piece_id=KM72SWE6TK7LLBMWBZTMGHA7VZYCPUEL6EE5AJYQSHTZXNB736OQ, seg_pos={0 0}, upload=503789]: Long tail cancellation due to optimal threshold
Dec 29 14:08:57.625479426 (574h39m57.496562461s) [seg_pos={0 0}, upload=503789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 14:08:57.625550862 (574h39m57.496633907s) [upload=503789]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:08:57.625556002 (574h39m57.496639047s) [upload=503789]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:08:57.928419950 (574h39m57.799502995s) [upload=503789]: Done uploading segments...
Dec 29 14:08:57.928438576 (574h39m57.799521611s) [upload=503789]: Upload finished. err: <nil>
Duration: 6.309360371s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [..X..X...X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XXX] duration:
30.002368682s trace:
https://jaeger.dev.storj.io/trace/d95840e5160db39
FAIL (2025-12-29 14:09:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: GR5O4ZCGOG4ZJCX6LUDBYNWY6QV7G5V4EFTR77E7ZXFFHAXVSA2Q) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: BUI3VAXETDPZLSKE6WWEB2RZHYCTURE6PJGUAWIL4QRSPVJBDWSQ) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: UXK5HV77MFD2VCPT4UJ2CHANYITS7MAHRLSZUTPTOV5O4ZFEOIAQ) context deadline exceeded
======LOGS======
segments...
Dec 29 14:08:54.984379378 (574h39m54.855462413s) [upload=503789]: Got next segment. Inline: false
Dec 29 14:08:54.984422079 (574h39m54.855505114s) [upload=503789]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 14:08:54.984428572 (574h39m54.855511607s) [upload=503789]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 14:08:55.232149008 (574h39m55.103232053s) [seg_pos={0 0}, upload=503789]: Begin upload segment...
Dec 29 14:08:55.232187962 (574h39m55.103271007s) [seg_pos={0 0}, upload=503789]: Done begin upload segment.
Dec 29 14:08:55.232216888 (574h39m55.103299943s) [noise=true , address=145.79.227.36:4519 , piece_id=FAM6EJKRUJKH45X5XA4IKSAGDSXJF2M2U2HWKVU7WSCKYINYRU6A, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.232253026 (574h39m55.103336071s) [noise=true , address=45.91.116.5:4299 , piece_id=UXK5HV77MFD2VCPT4UJ2CHANYITS7MAHRLSZUTPTOV5O4ZFEOIAQ, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.232620637 (574h39m55.103703672s) [noise=true , address=145.79.227.16:4911 , piece_id=6CPTKA4R6Z7NOCET76CFGO43MRQNBXAC5GEI5CAQSS4XN4DDQ5UA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.232629594 (574h39m55.103712629s) [noise=true , address=145.79.227.15:4875 , piece_id=ES2BZXFNLVSXZ3GJB6V7HL5O3L5ADI6OUW6XERBHF5PAX3AXHEWQ, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.232933292 (574h39m55.104016337s) [noise=true , address=77.90.31.14:4661 , piece_id=GR5O4ZCGOG4ZJCX6LUDBYNWY6QV7G5V4EFTR77E7ZXFFHAXVSA2Q, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233056236 (574h39m55.104139271s) [noise=true , address=77.90.31.21:4880 , piece_id=FDD6SMQQBEWH4MUKXTKXGWSTP6IDPEWRVZRDSTKLYZTPPNSN4W4Q, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233255676 (574h39m55.104338721s) [noise=true , address=45.91.116.4:4937 , piece_id=KM72SWE6TK7LLBMWBZTMGHA7VZYCPUEL6EE5AJYQSHTZXNB736OQ, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233345988 (574h39m55.104429033s) [noise=true , address=145.79.227.14:4839 , piece_id=BBKOWRIZJBDOCJKUY4WZR5SVS4AHLOGWNFFT33AE66NQCLLAUPEA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233538174 (574h39m55.104621209s) [noise=true , address=45.91.116.10:4084 , piece_id=WZGTISH7U57JMU27FPZDOR7JECEKX6BQ3P7SOXSQVNICDU2VAOUA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233719138 (574h39m55.104802173s) [noise=true , address=145.79.227.42:4123 , piece_id=4WKBVAFEMMRUG7ACQOAK2WCCJCRJVFZ52FCQR2AE7QA6YFWY7IVA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.233878271 (574h39m55.104961316s) [noise=true , address=45.91.116.13:4156 , piece_id=BUI3VAXETDPZLSKE6WWEB2RZHYCTURE6PJGUAWIL4QRSPVJBDWSQ, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.234172571 (574h39m55.105255606s) [noise=true , address=145.79.227.47:4265 , piece_id=G3VSWQDAVF3IUKVAYSZM4SOXZ7HWM2YV4CFMTNZKDPCWRZXZH5EA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.234295366 (574h39m55.105378411s) [noise=true , address=77.90.31.16:4735 , piece_id=RHAAEZJQ2DMYRLKC5BHWVNJ3ABKIAYVLYS3UJI5PH357F6QMNOPA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.234669588 (574h39m55.105752624s) [noise=true , address=145.79.227.39:4014 , piece_id=DKGPCCIHJWASXT6BOAYVJNKYKQDHKIEGQEXUJZLOSNRCI2ZFLHHA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.234740844 (574h39m55.105823889s) [noise=true , address=77.90.31.15:4697 , piece_id=ILUM2L5VOI47K7OVO6RKP3DAYXVWA6HEC7HBVHLGRJ4OLVPPDT7A, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:55.235331178 (574h39m55.106414223s) [noise=true , address=145.79.227.45:4231 , piece_id=CF5G3MSPJIXHEKLNOXCM4IIQAQGQORNGUNNW4NAQYUHZO4HTC2BA, seg_pos={0 0}, upload=503789]: Uploading piece...
Dec 29 14:08:56.935848616 (574h39m56.806931661s) [upload=503789]: Next returned nil segment
Dec 29 14:08:56.935862312 (574h39m56.806945347s) [upload=503789]: Waiting for error group managing segments...
Dec 29 14:08:57.216618483 (574h39m57.087701519s) [noise=true , address=145.79.227.39:4014 , piece_id=DKGPCCIHJWASXT6BOAYVJNKYKQDHKIEGQEXUJZLOSNRCI2ZFLHHA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.222446151 (574h39m57.093529196s) [noise=true , address=145.79.227.14:4839 , piece_id=BBKOWRIZJBDOCJKUY4WZR5SVS4AHLOGWNFFT33AE66NQCLLAUPEA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.222625933 (574h39m57.093708968s) [noise=true , address=145.79.227.15:4875 , piece_id=ES2BZXFNLVSXZ3GJB6V7HL5O3L5ADI6OUW6XERBHF5PAX3AXHEWQ, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.222951603 (574h39m57.094034648s) [noise=true , address=145.79.227.16:4911 , piece_id=6CPTKA4R6Z7NOCET76CFGO43MRQNBXAC5GEI5CAQSS4XN4DDQ5UA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.223015545 (574h39m57.09409858s) [noise=true , address=145.79.227.45:4231 , piece_id=CF5G3MSPJIXHEKLNOXCM4IIQAQGQORNGUNNW4NAQYUHZO4HTC2BA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.237053956 (574h39m57.108137001s) [noise=true , address=145.79.227.47:4265 , piece_id=G3VSWQDAVF3IUKVAYSZM4SOXZ7HWM2YV4CFMTNZKDPCWRZXZH5EA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.254897743 (574h39m57.125980778s) [noise=true , address=145.79.227.42:4123 , piece_id=4WKBVAFEMMRUG7ACQOAK2WCCJCRJVFZ52FCQR2AE7QA6YFWY7IVA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.274718345 (574h39m57.14580139s) [noise=true , address=145.79.227.36:4519 , piece_id=FAM6EJKRUJKH45X5XA4IKSAGDSXJF2M2U2HWKVU7WSCKYINYRU6A, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.445595220 (574h39m57.316678255s) [noise=true , address=77.90.31.14:4661 , piece_id=GR5O4ZCGOG4ZJCX6LUDBYNWY6QV7G5V4EFTR77E7ZXFFHAXVSA2Q, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.497711177 (574h39m57.368794222s) [noise=true , address=45.91.116.13:4156 , piece_id=BUI3VAXETDPZLSKE6WWEB2RZHYCTURE6PJGUAWIL4QRSPVJBDWSQ, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.501569313 (574h39m57.372652358s) [noise=true , address=45.91.116.5:4299 , piece_id=UXK5HV77MFD2VCPT4UJ2CHANYITS7MAHRLSZUTPTOV5O4ZFEOIAQ, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.509095373 (574h39m57.380178408s) [noise=true , address=77.90.31.16:4735 , piece_id=RHAAEZJQ2DMYRLKC5BHWVNJ3ABKIAYVLYS3UJI5PH357F6QMNOPA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.624926363 (574h39m57.496009398s) [noise=true , address=45.91.116.10:4084 , piece_id=WZGTISH7U57JMU27FPZDOR7JECEKX6BQ3P7SOXSQVNICDU2VAOUA, seg_pos={0 0}, upload=503789]: Done uploading piece. err: <nil>
Dec 29 14:08:57.624950658 (574h39m57.496033703s) [seg_pos={0 0}, upload=503789]: Segment reached optimal threshold of 13 pieces.
Dec 29 14:08:57.625146321 (574h39m57.496229356s) [noise=true , address=77.90.31.15:4697 , piece_id=ILUM2L5VOI47K7OVO6RKP3DAYXVWA6HEC7HBVHLGRJ4OLVPPDT7A, seg_pos={0 0}, upload=503789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:54892->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:54892->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54892->77.90.31.15:4697: use of closed network connection
Dec 29 14:08:57.625152713 (574h39m57.496235748s) [noise=true , address=77.90.31.15:4697 , piece_id=ILUM2L5VOI47K7OVO6RKP3DAYXVWA6HEC7HBVHLGRJ4OLVPPDT7A, seg_pos={0 0}, upload=503789]: Long tail cancellation due to optimal threshold
Dec 29 14:08:57.625327216 (574h39m57.496410261s) [noise=true , address=77.90.31.21:4880 , piece_id=FDD6SMQQBEWH4MUKXTKXGWSTP6IDPEWRVZRDSTKLYZTPPNSN4W4Q, seg_pos={0 0}, upload=503789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 29 14:08:57.625343327 (574h39m57.496426362s) [noise=true , address=77.90.31.21:4880 , piece_id=FDD6SMQQBEWH4MUKXTKXGWSTP6IDPEWRVZRDSTKLYZTPPNSN4W4Q, seg_pos={0 0}, upload=503789]: Long tail cancellation due to optimal threshold
Dec 29 14:08:57.625452785 (574h39m57.49653582s) [noise=true , address=45.91.116.4:4937 , piece_id=KM72SWE6TK7LLBMWBZTMGHA7VZYCPUEL6EE5AJYQSHTZXNB736OQ, seg_pos={0 0}, upload=503789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:48452->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:48452->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48452->45.91.116.4:4937: use of closed network connection
Dec 29 14:08:57.625463064 (574h39m57.496546099s) [noise=true , address=45.91.116.4:4937 , piece_id=KM72SWE6TK7LLBMWBZTMGHA7VZYCPUEL6EE5AJYQSHTZXNB736OQ, seg_pos={0 0}, upload=503789]: Long tail cancellation due to optimal threshold
Dec 29 14:08:57.625479426 (574h39m57.496562461s) [seg_pos={0 0}, upload=503789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 14:08:57.625550862 (574h39m57.496633907s) [upload=503789]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:08:57.625556002 (574h39m57.496639047s) [upload=503789]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:08:57.928419950 (574h39m57.799502995s) [upload=503789]: Done uploading segments...
Dec 29 14:08:57.928438576 (574h39m57.799521611s) [upload=503789]: Upload finished. err: <nil>
Duration: 30.002395514s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXXX] duration:
5.001040325s trace:
https://jaeger.dev.storj.io/trace/57650ad86ea17077
FAIL (2025-12-29 14:09:06 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001073339s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.....................X...X.X.XX....X........X.X............X] duration:
30.000985485s trace:
https://jaeger.dev.storj.io/trace/6c01aa8d1f60b948
FAIL (2025-12-29 14:08:58 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001025332s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [....................X...........X...X....X....X.XX..........] duration:
5.922310292s trace:
https://jaeger.dev.storj.io/trace/5dd0c6b12327bd0a
FAIL (2025-12-29 14:08:57 UTC)
orbiter warning: context deadline exceeded
Duration: 5.922338454s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.....................X...........X...X....X....X.XX.........] duration:
37.90423012s trace:
https://jaeger.dev.storj.io/trace/5dd0c6b12327bd0a
FAIL (2025-12-29 14:08:49 UTC)
orbiter warning: context deadline exceeded
Duration: 37.904560811s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [............................................XXX..X..XX......] duration:
1m0.002727305s trace:
https://jaeger.dev.storj.io/trace/266137264da4c90
FAIL (2025-12-29 14:08:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded
======LOGS======
NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment
Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments...
Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection
Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection
Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection
Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments...
Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil>
Duration: 1m0.002739628s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [...X..X...X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....XX] duration:
30.002691539s trace:
https://jaeger.dev.storj.io/trace/20a74b0779cbc650
FAIL (2025-12-29 14:08:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 6KYXT4WHJ7IUWCV7HUASQ7I324IL2J7K57NFS4NOZV6TNZBK6ECQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: QYE3J33NVVIIAIK2ACW323FHGVQLFWC5AOX2GMW7YLWCIOVUE6AA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: RVGXHC5BB3RY6GH5OZSVEVS5USVTHX3A7EL5WYVFW3CDDCJYRS4A) context deadline exceeded
======LOGS======
9 14:08:14.174171834 (574h39m14.045254869s) [upload=503779]: Starting upload
Dec 29 14:08:14.174252056 (574h39m14.045335101s) [upload=503779]: Done starting upload
Dec 29 14:08:14.174369090 (574h39m14.045452135s) [upload=503779]: Uploading segments...
Dec 29 14:08:14.174506181 (574h39m14.045589226s) [upload=503779]: Got next segment. Inline: false
Dec 29 14:08:14.174541277 (574h39m14.045624323s) [upload=503779]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 14:08:14.174546728 (574h39m14.045629773s) [upload=503779]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 14:08:14.422796320 (574h39m14.293879355s) [seg_pos={0 0}, upload=503779]: Begin upload segment...
Dec 29 14:08:14.422847068 (574h39m14.293930103s) [seg_pos={0 0}, upload=503779]: Done begin upload segment.
Dec 29 14:08:14.422891452 (574h39m14.293974487s) [noise=true , address=145.79.227.17:4972 , piece_id=ASHIJRSN2QJ47JNCAEZWWOKUUGLAGYDT5DFA43VDWZN7KYXF6SFQ, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.422933522 (574h39m14.294016557s) [noise=true , address=145.79.227.3:4151 , piece_id=EMUDLRUT2U4TT7WQX67NLIBOTO2JLIEQ6PCRRB2PBPHXXJ2ZG3NA, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.423435608 (574h39m14.294518653s) [noise=true , address=145.79.227.42:4122 , piece_id=AWMOJCYM6CTJ56CBNSR74PZLYLHPSCZRU3RPJ4FDVT5G3KQWP3KQ, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.423774543 (574h39m14.294857578s) [noise=true , address=45.91.116.5:4299 , piece_id=7DLFF7WREM6WXCMDFHU7DOO6OC6ACQ37TP2A6K3WKGBQW5GDFLMQ, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.424039969 (574h39m14.295123004s) [noise=true , address=45.91.116.13:4154 , piece_id=2B2UWDWOII6G5V3FYOJMPKPYSYW5YFE4U6GVK6JCJ7VO7TCHORJQ, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.424056471 (574h39m14.295139506s) [noise=true , address=45.91.116.14:4190 , piece_id=RVGXHC5BB3RY6GH5OZSVEVS5USVTHX3A7EL5WYVFW3CDDCJYRS4A, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.424314572 (574h39m14.295397607s) [noise=true , address=77.90.31.13:4626 , piece_id=7CLE7TWBBBBBLII4MPTUP7UE26HT5D4ZBFAUGD5PDJNCXLXSOQOA, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.424418239 (574h39m14.295501274s) [noise=true , address=145.79.227.46:4603 , piece_id=DHYLXLRIJEYZ4HSNFP3HCIO34PHD7UAOUD2MAFBPNP6MSKPGLUFQ, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.424552946 (574h39m14.295635981s) [noise=true , address=45.91.116.6:4975 , piece_id=6KYXT4WHJ7IUWCV7HUASQ7I324IL2J7K57NFS4NOZV6TNZBK6ECQ, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.424849540 (574h39m14.295932575s) [noise=true , address=145.79.227.7:4295 , piece_id=JL57OO3SFOQJ53TIWUGG2JDBFOPTV54BDIHA7SYKEHY2QXCOBWUA, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.424905687 (574h39m14.295988732s) [noise=true , address=77.90.31.14:4661 , piece_id=IX56Y6ZJDWGR44GBHSCMS7ZALPQQDNNTMVNPP6LLGJXLP3THDAWA, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.425121218 (574h39m14.296204253s) [noise=true , address=145.79.227.39:4015 , piece_id=4KYXKOKBQEAMRYLEPD427AELYCZ5I5TM3B4FPNBHHOK5WSIGOFIA, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.425271554 (574h39m14.296354589s) [noise=true , address=77.90.31.15:4700 , piece_id=QYE3J33NVVIIAIK2ACW323FHGVQLFWC5AOX2GMW7YLWCIOVUE6AA, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.425649523 (574h39m14.296732558s) [noise=true , address=77.90.31.21:4880 , piece_id=24WX2NOIUXILNQWCVDLM37SX7AHGSAAC33D546RGP5TSBB3ME4OQ, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.426128426 (574h39m14.297211461s) [noise=true , address=45.91.116.7:4009 , piece_id=7KPIEM6HULXGSSBDOXGVMSFRMYV5AFZS35FBMYSGEXRIHHZYCVRA, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:14.426467261 (574h39m14.297550296s) [noise=true , address=77.90.31.11:4554 , piece_id=ZCWKMRKODCZQP6KPJAZKWE36DBGDGHWZ3YZLJLYDCIQLJCDVJNNQ, seg_pos={0 0}, upload=503779]: Uploading piece...
Dec 29 14:08:16.123124232 (574h39m15.994207267s) [upload=503779]: Next returned nil segment
Dec 29 14:08:16.123151034 (574h39m15.994234069s) [upload=503779]: Waiting for error group managing segments...
Dec 29 14:08:16.398583978 (574h39m16.269667023s) [noise=true , address=145.79.227.17:4972 , piece_id=ASHIJRSN2QJ47JNCAEZWWOKUUGLAGYDT5DFA43VDWZN7KYXF6SFQ, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.412095285 (574h39m16.28317833s) [noise=true , address=145.79.227.39:4015 , piece_id=4KYXKOKBQEAMRYLEPD427AELYCZ5I5TM3B4FPNBHHOK5WSIGOFIA, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.412288594 (574h39m16.283371629s) [noise=true , address=145.79.227.46:4603 , piece_id=DHYLXLRIJEYZ4HSNFP3HCIO34PHD7UAOUD2MAFBPNP6MSKPGLUFQ, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.416283130 (574h39m16.287366175s) [noise=true , address=145.79.227.3:4151 , piece_id=EMUDLRUT2U4TT7WQX67NLIBOTO2JLIEQ6PCRRB2PBPHXXJ2ZG3NA, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.418633545 (574h39m16.28971659s) [noise=true , address=145.79.227.7:4295 , piece_id=JL57OO3SFOQJ53TIWUGG2JDBFOPTV54BDIHA7SYKEHY2QXCOBWUA, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.428753915 (574h39m16.29983696s) [noise=true , address=145.79.227.42:4122 , piece_id=AWMOJCYM6CTJ56CBNSR74PZLYLHPSCZRU3RPJ4FDVT5G3KQWP3KQ, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.675259989 (574h39m16.546343024s) [noise=true , address=77.90.31.15:4700 , piece_id=QYE3J33NVVIIAIK2ACW323FHGVQLFWC5AOX2GMW7YLWCIOVUE6AA, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.677569978 (574h39m16.548653024s) [noise=true , address=77.90.31.11:4554 , piece_id=ZCWKMRKODCZQP6KPJAZKWE36DBGDGHWZ3YZLJLYDCIQLJCDVJNNQ, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.677991120 (574h39m16.549074155s) [noise=true , address=45.91.116.14:4190 , piece_id=RVGXHC5BB3RY6GH5OZSVEVS5USVTHX3A7EL5WYVFW3CDDCJYRS4A, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.715910795 (574h39m16.58699383s) [noise=true , address=77.90.31.13:4626 , piece_id=7CLE7TWBBBBBLII4MPTUP7UE26HT5D4ZBFAUGD5PDJNCXLXSOQOA, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.784845756 (574h39m16.655928801s) [noise=true , address=77.90.31.14:4661 , piece_id=IX56Y6ZJDWGR44GBHSCMS7ZALPQQDNNTMVNPP6LLGJXLP3THDAWA, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.824211925 (574h39m16.695294961s) [noise=true , address=45.91.116.6:4975 , piece_id=6KYXT4WHJ7IUWCV7HUASQ7I324IL2J7K57NFS4NOZV6TNZBK6ECQ, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.991639954 (574h39m16.862722989s) [noise=true , address=77.90.31.21:4880 , piece_id=24WX2NOIUXILNQWCVDLM37SX7AHGSAAC33D546RGP5TSBB3ME4OQ, seg_pos={0 0}, upload=503779]: Done uploading piece. err: <nil>
Dec 29 14:08:16.991677305 (574h39m16.86276034s) [seg_pos={0 0}, upload=503779]: Segment reached optimal threshold of 13 pieces.
Dec 29 14:08:16.991903064 (574h39m16.862986119s) [noise=true , address=45.91.116.5:4299 , piece_id=7DLFF7WREM6WXCMDFHU7DOO6OC6ACQ37TP2A6K3WKGBQW5GDFLMQ, seg_pos={0 0}, upload=503779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF
Dec 29 14:08:16.991912281 (574h39m16.862995326s) [noise=true , address=45.91.116.5:4299 , piece_id=7DLFF7WREM6WXCMDFHU7DOO6OC6ACQ37TP2A6K3WKGBQW5GDFLMQ, seg_pos={0 0}, upload=503779]: Long tail cancellation due to optimal threshold
Dec 29 14:08:16.992042480 (574h39m16.863125525s) [noise=true , address=45.91.116.7:4009 , piece_id=7KPIEM6HULXGSSBDOXGVMSFRMYV5AFZS35FBMYSGEXRIHHZYCVRA, seg_pos={0 0}, upload=503779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 29 14:08:16.992048591 (574h39m16.863131636s) [noise=true , address=45.91.116.7:4009 , piece_id=7KPIEM6HULXGSSBDOXGVMSFRMYV5AFZS35FBMYSGEXRIHHZYCVRA, seg_pos={0 0}, upload=503779]: Long tail cancellation due to optimal threshold
Dec 29 14:08:16.992057799 (574h39m16.863140834s) [noise=true , address=45.91.116.13:4154 , piece_id=2B2UWDWOII6G5V3FYOJMPKPYSYW5YFE4U6GVK6JCJ7VO7TCHORJQ, seg_pos={0 0}, upload=503779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:48730->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:48730->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48730->45.91.116.13:4154: use of closed network connection
Dec 29 14:08:16.992064442 (574h39m16.863147487s) [noise=true , address=45.91.116.13:4154 , piece_id=2B2UWDWOII6G5V3FYOJMPKPYSYW5YFE4U6GVK6JCJ7VO7TCHORJQ, seg_pos={0 0}, upload=503779]: Long tail cancellation due to optimal threshold
Dec 29 14:08:16.992080251 (574h39m16.863163286s) [seg_pos={0 0}, upload=503779]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 14:08:16.992154633 (574h39m16.863237678s) [upload=503779]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:08:16.992190141 (574h39m16.863273186s) [upload=503779]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:08:17.293636288 (574h39m17.164719334s) [upload=503779]: Done uploading segments...
Dec 29 14:08:17.293646087 (574h39m17.164729132s) [upload=503779]: Upload finished. err: <nil>
Duration: 30.002715994s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [......................X...X.X.XX....X........X.X............] duration:
23.693411589s trace:
https://jaeger.dev.storj.io/trace/385292ec60d33ebf
FAIL (2025-12-29 14:08:26 UTC)
orbiter warning: context deadline exceeded
Duration: 23.693445774s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXXX] duration:
5.001473954s trace:
https://jaeger.dev.storj.io/trace/1ce0f3da90267ad6
FAIL (2025-12-29 14:08:15 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00149773s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [X...X..X...X...X.XXXX.X.XX.X.XXX.XXXXX.XXX.X..X.X..XXXX....X] duration:
30.002693502s trace:
https://jaeger.dev.storj.io/trace/1809393c23f2a4da
FAIL (2025-12-29 14:07:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: VS2ADASTOBK5AZODLDDR2VYGCJVDU7BYB5AR3RBQ4VVTB7GXVW4A) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: NM3MIAUU5TV7QOIZR4AAWBP7WEUY4OBUZJRO7VCOX3BM2IYCGDAQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VYOX67DRWOCQUQXFHLH3KAFVZSDY6EFG6LRG7G6FO3PUEGZMFOXA) context deadline exceeded
======LOGS======
29 14:07:26.852795923 (574h38m26.723878958s) [upload=503768]: Starting upload
Dec 29 14:07:26.852899190 (574h38m26.723982235s) [upload=503768]: Done starting upload
Dec 29 14:07:26.852983911 (574h38m26.724066966s) [upload=503768]: Uploading segments...
Dec 29 14:07:26.853230420 (574h38m26.724313465s) [upload=503768]: Got next segment. Inline: false
Dec 29 14:07:26.853288150 (574h38m26.724371185s) [upload=503768]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 14:07:26.853293811 (574h38m26.724376846s) [upload=503768]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 14:07:27.121176238 (574h38m26.992259283s) [seg_pos={0 0}, upload=503768]: Begin upload segment...
Dec 29 14:07:27.121236642 (574h38m26.992319687s) [seg_pos={0 0}, upload=503768]: Done begin upload segment.
Dec 29 14:07:27.121286638 (574h38m26.992369683s) [noise=true , address=145.79.227.11:4730 , piece_id=EDYT3CWWBF4ZKXOCSKL7OC3PZ6AK2K5T43LVAIDNSEVZQBL7EMIQ, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.121352363 (574h38m26.992435408s) [noise=true , address=145.79.227.52:4446 , piece_id=VON6OKKYZV5HXREILAB5BVTPK2NMD46SCJGHMNI7EKC2YID2RWFQ, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.121967604 (574h38m26.993050649s) [noise=true , address=45.91.116.6:4974 , piece_id=VYOX67DRWOCQUQXFHLH3KAFVZSDY6EFG6LRG7G6FO3PUEGZMFOXA, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.122259130 (574h38m26.993342175s) [noise=true , address=45.91.116.11:4117 , piece_id=MVBUDUHLTTV7ONPI6CZXXNM3U5HOODGUXKORWAXTBCW4UNFYA6XQ, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.122417741 (574h38m26.993500776s) [noise=true , address=145.79.227.12:4767 , piece_id=ZKSDRJQU5ZS7BUOWBDZVKX6KMZVCWBLRHRICBPNTYHINH53EWKHQ, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.123237763 (574h38m26.994320798s) [noise=true , address=45.91.116.12:4226 , piece_id=VS2ADASTOBK5AZODLDDR2VYGCJVDU7BYB5AR3RBQ4VVTB7GXVW4A, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.123499221 (574h38m26.994582256s) [noise=true , address=145.79.227.16:4910 , piece_id=C7EXX2B2FYJALUF3JRK4VRABYEII2UAARFNYAORFCND3HIR6QN4A, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.124218180 (574h38m26.995301215s) [noise=true , address=145.79.227.6:4258 , piece_id=O2REM2SZINF4TPVREKT3EJ3N225L5M3PUISSNQLN726EJVXL23XA, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.124842248 (574h38m26.995925283s) [noise=true , address=145.79.227.51:4410 , piece_id=T24AVHVQ4LCFRM3RK4A3QIX7GNHGQICMM53Z5RVXNCSPO4PQHRHQ, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.125485473 (574h38m26.996568518s) [noise=true , address=45.91.116.14:4189 , piece_id=NM3MIAUU5TV7QOIZR4AAWBP7WEUY4OBUZJRO7VCOX3BM2IYCGDAQ, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.125881847 (574h38m26.996964882s) [noise=true , address=145.79.227.19:4077 , piece_id=UJCGJJCE737LWUDV56WO3DF4RBBNSH2MROS6OIYVHKYIA76776QA, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.127006169 (574h38m26.998089204s) [noise=true , address=145.79.227.50:4375 , piece_id=GO4WNVDOIEEEYQJF7K5SIH7SB4TO4LHVN7GKQO3RHLZ5F34AY5VQ, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.127062415 (574h38m26.99814545s) [noise=true , address=77.90.31.19:4841 , piece_id=QHSIQAUGQPR7PUP7L2VBN26JFID56KGO7YLQEJFGO6OBN65C7LGA, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.127795842 (574h38m26.998878887s) [noise=true , address=77.90.31.14:4664 , piece_id=64LOWFUNM3OIVNBCKN4MRILPFQVZQJNWTA5NZCE5N5E7VEERXJZA, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.127853903 (574h38m26.998936948s) [noise=true , address=45.91.116.13:4156 , piece_id=7DBCAJVWMQRZI2XNURIY6EJCPWXQVUOBZN5KJJVHGS2XBC5IUGKQ, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:27.128471589 (574h38m26.999554624s) [noise=true , address=45.91.116.7:4012 , piece_id=KY7V6PB3ZA2IVH3P75ZZTNWAZ45C5WOBZOA5PV5NKADV3ITVE5OQ, seg_pos={0 0}, upload=503768]: Uploading piece...
Dec 29 14:07:28.733226886 (574h38m28.604309921s) [upload=503768]: Next returned nil segment
Dec 29 14:07:28.733255880 (574h38m28.604338915s) [upload=503768]: Waiting for error group managing segments...
Dec 29 14:07:29.114143175 (574h38m28.98522621s) [noise=true , address=145.79.227.12:4767 , piece_id=ZKSDRJQU5ZS7BUOWBDZVKX6KMZVCWBLRHRICBPNTYHINH53EWKHQ, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.118313456 (574h38m28.989396491s) [noise=true , address=145.79.227.51:4410 , piece_id=T24AVHVQ4LCFRM3RK4A3QIX7GNHGQICMM53Z5RVXNCSPO4PQHRHQ, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.118968764 (574h38m28.990051799s) [noise=true , address=145.79.227.50:4375 , piece_id=GO4WNVDOIEEEYQJF7K5SIH7SB4TO4LHVN7GKQO3RHLZ5F34AY5VQ, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.119612791 (574h38m28.990695836s) [noise=true , address=145.79.227.11:4730 , piece_id=EDYT3CWWBF4ZKXOCSKL7OC3PZ6AK2K5T43LVAIDNSEVZQBL7EMIQ, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.120362367 (574h38m28.991445403s) [noise=true , address=145.79.227.16:4910 , piece_id=C7EXX2B2FYJALUF3JRK4VRABYEII2UAARFNYAORFCND3HIR6QN4A, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.135689984 (574h38m29.006773019s) [noise=true , address=145.79.227.19:4077 , piece_id=UJCGJJCE737LWUDV56WO3DF4RBBNSH2MROS6OIYVHKYIA76776QA, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.135940691 (574h38m29.007023726s) [noise=true , address=145.79.227.6:4258 , piece_id=O2REM2SZINF4TPVREKT3EJ3N225L5M3PUISSNQLN726EJVXL23XA, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.139147357 (574h38m29.010230392s) [noise=true , address=145.79.227.52:4446 , piece_id=VON6OKKYZV5HXREILAB5BVTPK2NMD46SCJGHMNI7EKC2YID2RWFQ, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.391980558 (574h38m29.263063603s) [noise=true , address=45.91.116.14:4189 , piece_id=NM3MIAUU5TV7QOIZR4AAWBP7WEUY4OBUZJRO7VCOX3BM2IYCGDAQ, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.407084879 (574h38m29.278167914s) [noise=true , address=45.91.116.12:4226 , piece_id=VS2ADASTOBK5AZODLDDR2VYGCJVDU7BYB5AR3RBQ4VVTB7GXVW4A, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.410397847 (574h38m29.281480892s) [noise=true , address=45.91.116.13:4156 , piece_id=7DBCAJVWMQRZI2XNURIY6EJCPWXQVUOBZN5KJJVHGS2XBC5IUGKQ, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:29.517263916 (574h38m29.388346961s) [noise=true , address=45.91.116.7:4012 , piece_id=KY7V6PB3ZA2IVH3P75ZZTNWAZ45C5WOBZOA5PV5NKADV3ITVE5OQ, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:30.142893415 (574h38m30.01397646s) [noise=true , address=45.91.116.6:4974 , piece_id=VYOX67DRWOCQUQXFHLH3KAFVZSDY6EFG6LRG7G6FO3PUEGZMFOXA, seg_pos={0 0}, upload=503768]: Done uploading piece. err: <nil>
Dec 29 14:07:30.142921028 (574h38m30.014004083s) [seg_pos={0 0}, upload=503768]: Segment reached optimal threshold of 13 pieces.
Dec 29 14:07:30.143244794 (574h38m30.014327839s) [noise=true , address=77.90.31.14:4664 , piece_id=64LOWFUNM3OIVNBCKN4MRILPFQVZQJNWTA5NZCE5N5E7VEERXJZA, seg_pos={0 0}, upload=503768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF
Dec 29 14:07:30.143254322 (574h38m30.014337377s) [noise=true , address=77.90.31.14:4664 , piece_id=64LOWFUNM3OIVNBCKN4MRILPFQVZQJNWTA5NZCE5N5E7VEERXJZA, seg_pos={0 0}, upload=503768]: Long tail cancellation due to optimal threshold
Dec 29 14:07:30.143273499 (574h38m30.014356544s) [noise=true , address=77.90.31.19:4841 , piece_id=QHSIQAUGQPR7PUP7L2VBN26JFID56KGO7YLQEJFGO6OBN65C7LGA, seg_pos={0 0}, upload=503768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:50356->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:50356->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50356->77.90.31.19:4841: use of closed network connection
Dec 29 14:07:30.143289810 (574h38m30.014372855s) [noise=true , address=77.90.31.19:4841 , piece_id=QHSIQAUGQPR7PUP7L2VBN26JFID56KGO7YLQEJFGO6OBN65C7LGA, seg_pos={0 0}, upload=503768]: Long tail cancellation due to optimal threshold
Dec 29 14:07:30.143479281 (574h38m30.014562316s) [noise=true , address=45.91.116.11:4117 , piece_id=MVBUDUHLTTV7ONPI6CZXXNM3U5HOODGUXKORWAXTBCW4UNFYA6XQ, seg_pos={0 0}, upload=503768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 29 14:07:30.143485703 (574h38m30.014568748s) [noise=true , address=45.91.116.11:4117 , piece_id=MVBUDUHLTTV7ONPI6CZXXNM3U5HOODGUXKORWAXTBCW4UNFYA6XQ, seg_pos={0 0}, upload=503768]: Long tail cancellation due to optimal threshold
Dec 29 14:07:30.143501863 (574h38m30.014584898s) [seg_pos={0 0}, upload=503768]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 14:07:30.143577227 (574h38m30.014660263s) [upload=503768]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:07:30.143581786 (574h38m30.014664831s) [upload=503768]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:07:30.434258245 (574h38m30.30534129s) [upload=503768]: Done uploading segments...
Dec 29 14:07:30.434282051 (574h38m30.305365096s) [upload=503768]: Upload finished. err: <nil>
Duration: 30.002714924s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXXX] duration:
5.001087711s trace:
https://jaeger.dev.storj.io/trace/4e0a5625d0e4657
FAIL (2025-12-29 14:07:49 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001112347s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............................................XXX..X..XX.....] duration:
55.082205281s trace:
https://jaeger.dev.storj.io/trace/266137264da4c90
FAIL (2025-12-29 14:07:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12d7KuNgPYL5Ca2MPKQQCueQNSsYQifHXMhQ87VEXUhrKpLjuks, Piece ID: GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA) context deadline exceeded; piecestore: (Node ID: 14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, Piece ID: GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ) context deadline exceeded; piecestore: (Node ID: 16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ, Piece ID: VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ) context deadline exceeded
======LOGS======
NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456681578 (574h33m8.327764633s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.456980868 (574h33m8.328063903s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457107550 (574h33m8.328190585s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457371402 (574h33m8.328454437s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457525796 (574h33m8.328608841s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.457736046 (574h33m8.328819081s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458001461 (574h33m8.329084496s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458310710 (574h33m8.329393745s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458539436 (574h33m8.329622481s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458714419 (574h33m8.329797454s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.458979544 (574h33m8.330062589s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459084343 (574h33m8.330167378s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:08.459551082 (574h33m8.330634117s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Uploading piece...
Dec 29 14:02:09.814952577 (574h33m9.686035612s) [upload=503692]: Next returned nil segment
Dec 29 14:02:09.814968837 (574h33m9.686051872s) [upload=503692]: Waiting for error group managing segments...
Dec 29 14:02:10.331300185 (574h33m10.20238323s) [noise=true , address=145.79.227.4:4195 , piece_id=TIUER26XCBVHPWXVN25UQR3ZLFPNI2JMWZQHPQGYGP3M6Y7HCUPQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.363584750 (574h33m10.234667785s) [noise=true , address=145.79.227.17:4002 , piece_id=K6KLMPYG2MZRI7ZZ5VJ4BCPG3ITTQFLYQXCBYBGREEC6JH22FGNA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.490410834 (574h33m10.361493869s) [noise=true , address=64.49.13.139:4476 , piece_id=EXRMMBOFS4VBCJC7GR66EK4GDMSY3OOZUEEE5RWOWAAHSKGKD6KQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.511054292 (574h33m10.382137337s) [noise=true , address=77.90.31.14:4694 , piece_id=BTZRGM4KCGS43QOTTL2MYVIDV7BBKCKPDPIJOLIBAJYQYUMKNGAA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.573772276 (574h33m10.444855321s) [noise=true , address=77.90.31.19:4874 , piece_id=GF2I6GXJRF4UZX7BUHPF6ZGRGX6XZYV3DSL5K2GOMNHYMWT4IXVQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:10.656808880 (574h33m10.527891925s) [noise=true , address=45.91.116.14:4223 , piece_id=2ZWIHM7XVC2F6U6I3U6OAR2LEU7NPJ5Y47NVP35S4ARCSWBJCGBQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.097603197 (574h33m10.968686232s) [noise=true , address=64.49.13.170:4720 , piece_id=2ZT7R66ZPMGZX366EKZH2K2LOBXFQNO7ONOGRV4KZ4JAYQXLTI2Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.179772278 (574h33m11.050855323s) [noise=true , address=145.79.227.30:4768 , piece_id=6PF2M6BI3PPCOHOW5CJDS2A4KYZYXSDUOG6ZROOA2XDAPDYUOQEQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.251126207 (574h33m11.122209252s) [noise=true , address=145.79.227.44:4211 , piece_id=Y4M2MZK2GJHWQ7VVP4AITNHLQEM43K7EJNZ66LHXBEY7LQ3VQ44Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.255745833 (574h33m11.126828869s) [noise=true , address=145.79.227.29:4947 , piece_id=GMSNTBHF6CHF6DF6VCH7KWC76N5LPKDNTRMXIWZC6FJ4QQK3KNBA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.270245643 (574h33m11.141328688s) [noise=true , address=193.149.164.143:4697 , piece_id=VCB4GDLS4MR7OYVBRCU2GWVI2GE5FTUXTZQACNGSVME5X4E37RRQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.272416647 (574h33m11.143499682s) [noise=true , address=145.79.227.40:4084 , piece_id=PSJAZTJK7SWNDA7YH3NGG4N673JQM6OURXMKJ377RQM46EZMFYEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.293768324 (574h33m11.164851379s) [noise=true , address=45.91.116.9:4294 , piece_id=GPDXDMHXH3CHPRNNBKGCDRUXNH3MHP3HCR4YDLFDDY72MXDMGFEA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426804324 (574h33m11.297887369s) [noise=true , address=145.79.227.34:4937 , piece_id=7LA5ANNVAAQ3MNO6SCENULKMU4AG7XUO2JKAJ53K26HF7DLM4JGA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: <nil>
Dec 29 14:02:11.426832397 (574h33m11.297915432s) [seg_pos={0 0}, upload=503692]: Segment reached optimal threshold of 14 pieces.
Dec 29 14:02:11.427039632 (574h33m11.298122667s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b): protocol: expected piece hash; context canceled; EOF
Dec 29 14:02:11.427046174 (574h33m11.298129209s) [noise=true , address=145.79.227.33:4879 , piece_id=QJIAECZK7JCHJVYUB4NK5O2WQMZ3BZQ24IK3RAZDGW3L3XAZGG7A, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427246265 (574h33m11.2983293s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1278Hwz624uAkvSY2xXZL7NTGcbgEKaPpJuBKp5cC53SW9g4HfQ): protocol: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33334->77.90.31.17:4776: use of closed network connection
Dec 29 14:02:11.427252577 (574h33m11.298335622s) [noise=true , address=77.90.31.17:4776 , piece_id=TXE7KFZQUKEXISFW7IIRUJYZQWQNBCSNNL72NBJ43FOPG7DSFRUA, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427277655 (574h33m11.2983607s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hkdk6W2VmYLGra1GpBVJjkypiydor2f1LDjjGL2Kkb6fJqENZ): protocol: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43778->77.90.31.10:4547: use of closed network connection
Dec 29 14:02:11.427347709 (574h33m11.298430744s) [noise=true , address=77.90.31.10:4547 , piece_id=NUVRW7SESS7FGPJRQ7CTIIHOWNVT4RHDVMMM6MPVRVU3XJL7AE5Q, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427386502 (574h33m11.298469607s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33498->64.49.13.169:4707: use of closed network connection
Dec 29 14:02:11.427436338 (574h33m11.298519383s) [noise=true , address=64.49.13.169:4707 , piece_id=FD5PJURCFU2XJCAROHRYDNMU5CTO2PBIINCNBOCVVZTBUVKNJGDQ, seg_pos={0 0}, upload=503692]: Long tail cancellation due to optimal threshold
Dec 29 14:02:11.427467718 (574h33m11.298550763s) [seg_pos={0 0}, upload=503692]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 14:02:11.427748081 (574h33m11.298831126s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 14:02:11.427756908 (574h33m11.298839943s) [upload=503692]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 14:02:11.723193732 (574h33m11.594276777s) [upload=503692]: Done uploading segments...
Dec 29 14:02:11.723262573 (574h33m11.594345618s) [upload=503692]: Upload finished. err: <nil>
Duration: 55.082233535s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [......................X...........X...X....X....X.XX........] duration:
13.326644222s trace:
https://jaeger.dev.storj.io/trace/5dd0c6b12327bd0a
FAIL (2025-12-29 14:07:37 UTC)
orbiter warning: context deadline exceeded
Duration: 13.326670291s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.......................X...X.X.XX....X........X.X...........] duration:
28.106604378s trace:
https://jaeger.dev.storj.io/trace/385292ec60d33ebf
FAIL (2025-12-29 14:07:35 UTC)
orbiter warning: context deadline exceeded
Duration: 28.106633674s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
55 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX.XXXXXXXXXX.XXXXXXX.XXX.XXXXXXXXXX.XXXXXXX] duration:
5.001674856s trace:
https://jaeger.dev.storj.io/trace/429731d70efd7953
FAIL (2025-12-29 14:07:14 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00169805s
core
uplink
us-select-1
us1-placement-20
|