Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...............................X.....X...X.X.....X..........] duration:
13.444620436s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:21:29 UTC)
orbiter warning: context deadline exceeded
Duration: 13.444640393s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
37.620977276s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:21:15 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 37.621004178s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.....X.........................................XX.X...XXX...] duration:
3.578106126s trace:
https://jaeger.dev.storj.io/trace/76e318e1b1f8473a
FAIL (2025-12-31 13:21:12 UTC)
orbiter warning: context deadline exceeded
Duration: 3.578156411s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................................X.....X...X.X.....X.........] duration:
23.462274025s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:21:03 UTC)
orbiter warning: context deadline exceeded
Duration: 23.462303302s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [......X.X...X.X........XX..X.X..X....XX..XX...X..X.XX.....X.] duration:
5.797093998s trace:
https://jaeger.dev.storj.io/trace/2155cb44a34c1d6a
FAIL (2025-12-31 13:20:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: ZPXJOW5DPWJYDAURNOZSCADXWL2PLJT55PSKM7FFTDOLCTS27ZIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: ANUMCSNKTCA3UZFM6LUWSF6QDGLQML6GODRWFWR5TISDLDUKNE5A) context deadline exceeded
======LOGS======
Dec 31 13:20:04.909573055 (621h51m4.780656101s) [upload=544917]: Starting upload
Dec 31 13:20:04.909682975 (621h51m4.78076602s) [upload=544917]: Done starting upload
Dec 31 13:20:04.909737949 (621h51m4.780821004s) [upload=544917]: Uploading segments...
Dec 31 13:20:04.909960884 (621h51m4.781043969s) [upload=544917]: Got next segment. Inline: false
Dec 31 13:20:04.910007552 (621h51m4.781090607s) [upload=544917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:20:04.910015878 (621h51m4.781098943s) [upload=544917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:20:05.153462984 (621h51m5.024546029s) [seg_pos={0 0}, upload=544917]: Begin upload segment...
Dec 31 13:20:05.153520053 (621h51m5.024603098s) [seg_pos={0 0}, upload=544917]: Done begin upload segment.
Dec 31 13:20:05.153556001 (621h51m5.024639046s) [noise=true , address=145.79.227.20:4439 , piece_id=TUEGJVUPEEHVAQCVR75R5EASZF6PSQZTZONOMK5BOBPS6TYG2NBA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.153567884 (621h51m5.024650929s) [noise=true , address=77.90.31.17:4770 , piece_id=ZPXJOW5DPWJYDAURNOZSCADXWL2PLJT55PSKM7FFTDOLCTS27ZIA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.154145514 (621h51m5.025228559s) [noise=true , address=145.79.227.38:4978 , piece_id=AWZD4EG35ARPSC6QTLKBEQIEBBTMGGNZABZVCBCKFH7KCFVIKJGA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.154720749 (621h51m5.025803794s) [noise=true , address=145.79.227.14:4839 , piece_id=AI7IRABIFSQV5WLO2BE32LRILS6X5253ZHXQEWNP6EAEWQLT6X4Q, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.155243354 (621h51m5.026326409s) [noise=true , address=45.91.116.7:4010 , piece_id=ANUMCSNKTCA3UZFM6LUWSF6QDGLQML6GODRWFWR5TISDLDUKNE5A, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.155929421 (621h51m5.027012466s) [noise=true , address=145.79.227.7:4293 , piece_id=PGKK4AJKSUH5422C7GYPJX4K6T7MR7AJ4KPTIOQA6TPQBPLVGR5A, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.158582382 (621h51m5.029665427s) [noise=true , address=77.90.31.19:4844 , piece_id=3EM5UKP234FHLWQKD2VXM3W57DMNVN3DPZGVOL2VW7LBCC4N5EVQ, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.160358183 (621h51m5.031441219s) [noise=true , address=145.79.227.45:4230 , piece_id=SL543O2M4LYUVDOLKVBHMNSRMCBPGH62L4TJQTNXRYLTHWCLONJA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.161896663 (621h51m5.032979718s) [noise=true , address=77.90.31.13:4627 , piece_id=25JC7E35BCNWHDT6XRL25W6DCFAOOBDECHOY2OX445QG4SYWYN2Q, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.164225217 (621h51m5.035308262s) [noise=true , address=145.79.227.43:4158 , piece_id=LRX67YOW6E7N6NOAMMCVPQDST3UMQKXSJNYR33PF7QYS462MSHJQ, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.167343213 (621h51m5.038426259s) [noise=true , address=145.79.227.11:4729 , piece_id=O37SVLEMHHRVJSGEEUR6BJ3E3PFSZM2GUZ4YITJNWFT6AYYDBHEA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.169132841 (621h51m5.040215876s) [noise=true , address=145.79.227.15:4876 , piece_id=I3OSJPPQXXCJU5N72NPJMQIQFV7JZZEIUWBGBRKAFSFOQACOMQZQ, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.170632877 (621h51m5.041715922s) [noise=true , address=145.79.227.19:4078 , piece_id=DHLITUUXA3XNRVTXDDJFNT44OMGOVVEQNNVJV7VXK7YLIMKRQP4A, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.172195302 (621h51m5.043278347s) [noise=true , address=45.91.116.8:4045 , piece_id=SEHDRY3YBREELBFJSUFVXCNY2EQN6UO7MP23YP2F6DI34FGJV5LA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.174220058 (621h51m5.045303103s) [noise=true , address=45.91.116.4:4940 , piece_id=CHRK6R7SVFJW2D3U6PU5NNI3VVRMZONUPBIHSMHIRDOIROEFE46A, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.176339083 (621h51m5.047422128s) [noise=true , address=145.79.227.41:4087 , piece_id=NG54TOXOYMCFGWNUD43PRMNTN4U3NE65UBOTOZDAUG54W7L7DLAA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:06.856419606 (621h51m6.727502651s) [upload=544917]: Next returned nil segment
Dec 31 13:20:06.856436408 (621h51m6.727519454s) [upload=544917]: Waiting for error group managing segments...
Dec 31 13:20:07.141556674 (621h51m7.012639719s) [noise=true , address=145.79.227.7:4293 , piece_id=PGKK4AJKSUH5422C7GYPJX4K6T7MR7AJ4KPTIOQA6TPQBPLVGR5A, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.142562389 (621h51m7.013645424s) [noise=true , address=145.79.227.38:4978 , piece_id=AWZD4EG35ARPSC6QTLKBEQIEBBTMGGNZABZVCBCKFH7KCFVIKJGA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.145058963 (621h51m7.016142009s) [noise=true , address=145.79.227.14:4839 , piece_id=AI7IRABIFSQV5WLO2BE32LRILS6X5253ZHXQEWNP6EAEWQLT6X4Q, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.156166072 (621h51m7.027249107s) [noise=true , address=145.79.227.45:4230 , piece_id=SL543O2M4LYUVDOLKVBHMNSRMCBPGH62L4TJQTNXRYLTHWCLONJA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.166040996 (621h51m7.037124041s) [noise=true , address=145.79.227.19:4078 , piece_id=DHLITUUXA3XNRVTXDDJFNT44OMGOVVEQNNVJV7VXK7YLIMKRQP4A, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.167662463 (621h51m7.038745498s) [noise=true , address=145.79.227.41:4087 , piece_id=NG54TOXOYMCFGWNUD43PRMNTN4U3NE65UBOTOZDAUG54W7L7DLAA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.186550749 (621h51m7.057633794s) [noise=true , address=145.79.227.15:4876 , piece_id=I3OSJPPQXXCJU5N72NPJMQIQFV7JZZEIUWBGBRKAFSFOQACOMQZQ, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.204126385 (621h51m7.07520943s) [noise=true , address=145.79.227.20:4439 , piece_id=TUEGJVUPEEHVAQCVR75R5EASZF6PSQZTZONOMK5BOBPS6TYG2NBA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.246668731 (621h51m7.117751766s) [noise=true , address=145.79.227.11:4729 , piece_id=O37SVLEMHHRVJSGEEUR6BJ3E3PFSZM2GUZ4YITJNWFT6AYYDBHEA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.355706834 (621h51m7.226789879s) [noise=true , address=77.90.31.17:4770 , piece_id=ZPXJOW5DPWJYDAURNOZSCADXWL2PLJT55PSKM7FFTDOLCTS27ZIA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.373898003 (621h51m7.244981038s) [noise=true , address=77.90.31.13:4627 , piece_id=25JC7E35BCNWHDT6XRL25W6DCFAOOBDECHOY2OX445QG4SYWYN2Q, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.468763709 (621h51m7.339846744s) [noise=true , address=45.91.116.7:4010 , piece_id=ANUMCSNKTCA3UZFM6LUWSF6QDGLQML6GODRWFWR5TISDLDUKNE5A, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.487324702 (621h51m7.358407747s) [noise=true , address=45.91.116.4:4940 , piece_id=CHRK6R7SVFJW2D3U6PU5NNI3VVRMZONUPBIHSMHIRDOIROEFE46A, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.487349278 (621h51m7.358432314s) [seg_pos={0 0}, upload=544917]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:20:07.487582653 (621h51m7.358665698s) [noise=true , address=45.91.116.8:4045 , piece_id=SEHDRY3YBREELBFJSUFVXCNY2EQN6UO7MP23YP2F6DI34FGJV5LA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 31 13:20:07.487591129 (621h51m7.358674164s) [noise=true , address=45.91.116.8:4045 , piece_id=SEHDRY3YBREELBFJSUFVXCNY2EQN6UO7MP23YP2F6DI34FGJV5LA, seg_pos={0 0}, upload=544917]: Long tail cancellation due to optimal threshold
Dec 31 13:20:07.488016559 (621h51m7.359099614s) [noise=true , address=77.90.31.19:4844 , piece_id=3EM5UKP234FHLWQKD2VXM3W57DMNVN3DPZGVOL2VW7LBCC4N5EVQ, seg_pos={0 0}, upload=544917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:37848->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:37848->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37848->77.90.31.19:4844: use of closed network connection
Dec 31 13:20:07.488039884 (621h51m7.359122929s) [noise=true , address=77.90.31.19:4844 , piece_id=3EM5UKP234FHLWQKD2VXM3W57DMNVN3DPZGVOL2VW7LBCC4N5EVQ, seg_pos={0 0}, upload=544917]: Long tail cancellation due to optimal threshold
Dec 31 13:20:07.488259471 (621h51m7.359342517s) [noise=true , address=145.79.227.43:4158 , piece_id=LRX67YOW6E7N6NOAMMCVPQDST3UMQKXSJNYR33PF7QYS462MSHJQ, seg_pos={0 0}, upload=544917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 31 13:20:07.488281163 (621h51m7.359364208s) [noise=true , address=145.79.227.43:4158 , piece_id=LRX67YOW6E7N6NOAMMCVPQDST3UMQKXSJNYR33PF7QYS462MSHJQ, seg_pos={0 0}, upload=544917]: Long tail cancellation due to optimal threshold
Dec 31 13:20:07.488317883 (621h51m7.359400918s) [seg_pos={0 0}, upload=544917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:20:07.488459302 (621h51m7.359542347s) [upload=544917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:20:07.488466656 (621h51m7.359549701s) [upload=544917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:20:07.772371950 (621h51m7.643454985s) [upload=544917]: Done uploading segments...
Dec 31 13:20:07.772392689 (621h51m7.643475724s) [upload=544917]: Upload finished. err: <nil>
Duration: 5.797110209s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X......X.X...X.X........XX..X.X..X....XX..XX...X..X.XX.....X] duration:
30.003592724s trace:
https://jaeger.dev.storj.io/trace/2155cb44a34c1d6a
FAIL (2025-12-31 13:20:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, Piece ID: ZPXJOW5DPWJYDAURNOZSCADXWL2PLJT55PSKM7FFTDOLCTS27ZIA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: ANUMCSNKTCA3UZFM6LUWSF6QDGLQML6GODRWFWR5TISDLDUKNE5A) context deadline exceeded
======LOGS======
Dec 31 13:20:04.909573055 (621h51m4.780656101s) [upload=544917]: Starting upload
Dec 31 13:20:04.909682975 (621h51m4.78076602s) [upload=544917]: Done starting upload
Dec 31 13:20:04.909737949 (621h51m4.780821004s) [upload=544917]: Uploading segments...
Dec 31 13:20:04.909960884 (621h51m4.781043969s) [upload=544917]: Got next segment. Inline: false
Dec 31 13:20:04.910007552 (621h51m4.781090607s) [upload=544917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:20:04.910015878 (621h51m4.781098943s) [upload=544917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:20:05.153462984 (621h51m5.024546029s) [seg_pos={0 0}, upload=544917]: Begin upload segment...
Dec 31 13:20:05.153520053 (621h51m5.024603098s) [seg_pos={0 0}, upload=544917]: Done begin upload segment.
Dec 31 13:20:05.153556001 (621h51m5.024639046s) [noise=true , address=145.79.227.20:4439 , piece_id=TUEGJVUPEEHVAQCVR75R5EASZF6PSQZTZONOMK5BOBPS6TYG2NBA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.153567884 (621h51m5.024650929s) [noise=true , address=77.90.31.17:4770 , piece_id=ZPXJOW5DPWJYDAURNOZSCADXWL2PLJT55PSKM7FFTDOLCTS27ZIA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.154145514 (621h51m5.025228559s) [noise=true , address=145.79.227.38:4978 , piece_id=AWZD4EG35ARPSC6QTLKBEQIEBBTMGGNZABZVCBCKFH7KCFVIKJGA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.154720749 (621h51m5.025803794s) [noise=true , address=145.79.227.14:4839 , piece_id=AI7IRABIFSQV5WLO2BE32LRILS6X5253ZHXQEWNP6EAEWQLT6X4Q, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.155243354 (621h51m5.026326409s) [noise=true , address=45.91.116.7:4010 , piece_id=ANUMCSNKTCA3UZFM6LUWSF6QDGLQML6GODRWFWR5TISDLDUKNE5A, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.155929421 (621h51m5.027012466s) [noise=true , address=145.79.227.7:4293 , piece_id=PGKK4AJKSUH5422C7GYPJX4K6T7MR7AJ4KPTIOQA6TPQBPLVGR5A, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.158582382 (621h51m5.029665427s) [noise=true , address=77.90.31.19:4844 , piece_id=3EM5UKP234FHLWQKD2VXM3W57DMNVN3DPZGVOL2VW7LBCC4N5EVQ, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.160358183 (621h51m5.031441219s) [noise=true , address=145.79.227.45:4230 , piece_id=SL543O2M4LYUVDOLKVBHMNSRMCBPGH62L4TJQTNXRYLTHWCLONJA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.161896663 (621h51m5.032979718s) [noise=true , address=77.90.31.13:4627 , piece_id=25JC7E35BCNWHDT6XRL25W6DCFAOOBDECHOY2OX445QG4SYWYN2Q, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.164225217 (621h51m5.035308262s) [noise=true , address=145.79.227.43:4158 , piece_id=LRX67YOW6E7N6NOAMMCVPQDST3UMQKXSJNYR33PF7QYS462MSHJQ, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.167343213 (621h51m5.038426259s) [noise=true , address=145.79.227.11:4729 , piece_id=O37SVLEMHHRVJSGEEUR6BJ3E3PFSZM2GUZ4YITJNWFT6AYYDBHEA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.169132841 (621h51m5.040215876s) [noise=true , address=145.79.227.15:4876 , piece_id=I3OSJPPQXXCJU5N72NPJMQIQFV7JZZEIUWBGBRKAFSFOQACOMQZQ, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.170632877 (621h51m5.041715922s) [noise=true , address=145.79.227.19:4078 , piece_id=DHLITUUXA3XNRVTXDDJFNT44OMGOVVEQNNVJV7VXK7YLIMKRQP4A, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.172195302 (621h51m5.043278347s) [noise=true , address=45.91.116.8:4045 , piece_id=SEHDRY3YBREELBFJSUFVXCNY2EQN6UO7MP23YP2F6DI34FGJV5LA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.174220058 (621h51m5.045303103s) [noise=true , address=45.91.116.4:4940 , piece_id=CHRK6R7SVFJW2D3U6PU5NNI3VVRMZONUPBIHSMHIRDOIROEFE46A, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:05.176339083 (621h51m5.047422128s) [noise=true , address=145.79.227.41:4087 , piece_id=NG54TOXOYMCFGWNUD43PRMNTN4U3NE65UBOTOZDAUG54W7L7DLAA, seg_pos={0 0}, upload=544917]: Uploading piece...
Dec 31 13:20:06.856419606 (621h51m6.727502651s) [upload=544917]: Next returned nil segment
Dec 31 13:20:06.856436408 (621h51m6.727519454s) [upload=544917]: Waiting for error group managing segments...
Dec 31 13:20:07.141556674 (621h51m7.012639719s) [noise=true , address=145.79.227.7:4293 , piece_id=PGKK4AJKSUH5422C7GYPJX4K6T7MR7AJ4KPTIOQA6TPQBPLVGR5A, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.142562389 (621h51m7.013645424s) [noise=true , address=145.79.227.38:4978 , piece_id=AWZD4EG35ARPSC6QTLKBEQIEBBTMGGNZABZVCBCKFH7KCFVIKJGA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.145058963 (621h51m7.016142009s) [noise=true , address=145.79.227.14:4839 , piece_id=AI7IRABIFSQV5WLO2BE32LRILS6X5253ZHXQEWNP6EAEWQLT6X4Q, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.156166072 (621h51m7.027249107s) [noise=true , address=145.79.227.45:4230 , piece_id=SL543O2M4LYUVDOLKVBHMNSRMCBPGH62L4TJQTNXRYLTHWCLONJA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.166040996 (621h51m7.037124041s) [noise=true , address=145.79.227.19:4078 , piece_id=DHLITUUXA3XNRVTXDDJFNT44OMGOVVEQNNVJV7VXK7YLIMKRQP4A, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.167662463 (621h51m7.038745498s) [noise=true , address=145.79.227.41:4087 , piece_id=NG54TOXOYMCFGWNUD43PRMNTN4U3NE65UBOTOZDAUG54W7L7DLAA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.186550749 (621h51m7.057633794s) [noise=true , address=145.79.227.15:4876 , piece_id=I3OSJPPQXXCJU5N72NPJMQIQFV7JZZEIUWBGBRKAFSFOQACOMQZQ, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.204126385 (621h51m7.07520943s) [noise=true , address=145.79.227.20:4439 , piece_id=TUEGJVUPEEHVAQCVR75R5EASZF6PSQZTZONOMK5BOBPS6TYG2NBA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.246668731 (621h51m7.117751766s) [noise=true , address=145.79.227.11:4729 , piece_id=O37SVLEMHHRVJSGEEUR6BJ3E3PFSZM2GUZ4YITJNWFT6AYYDBHEA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.355706834 (621h51m7.226789879s) [noise=true , address=77.90.31.17:4770 , piece_id=ZPXJOW5DPWJYDAURNOZSCADXWL2PLJT55PSKM7FFTDOLCTS27ZIA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.373898003 (621h51m7.244981038s) [noise=true , address=77.90.31.13:4627 , piece_id=25JC7E35BCNWHDT6XRL25W6DCFAOOBDECHOY2OX445QG4SYWYN2Q, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.468763709 (621h51m7.339846744s) [noise=true , address=45.91.116.7:4010 , piece_id=ANUMCSNKTCA3UZFM6LUWSF6QDGLQML6GODRWFWR5TISDLDUKNE5A, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.487324702 (621h51m7.358407747s) [noise=true , address=45.91.116.4:4940 , piece_id=CHRK6R7SVFJW2D3U6PU5NNI3VVRMZONUPBIHSMHIRDOIROEFE46A, seg_pos={0 0}, upload=544917]: Done uploading piece. err: <nil>
Dec 31 13:20:07.487349278 (621h51m7.358432314s) [seg_pos={0 0}, upload=544917]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:20:07.487582653 (621h51m7.358665698s) [noise=true , address=45.91.116.8:4045 , piece_id=SEHDRY3YBREELBFJSUFVXCNY2EQN6UO7MP23YP2F6DI34FGJV5LA, seg_pos={0 0}, upload=544917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 31 13:20:07.487591129 (621h51m7.358674164s) [noise=true , address=45.91.116.8:4045 , piece_id=SEHDRY3YBREELBFJSUFVXCNY2EQN6UO7MP23YP2F6DI34FGJV5LA, seg_pos={0 0}, upload=544917]: Long tail cancellation due to optimal threshold
Dec 31 13:20:07.488016559 (621h51m7.359099614s) [noise=true , address=77.90.31.19:4844 , piece_id=3EM5UKP234FHLWQKD2VXM3W57DMNVN3DPZGVOL2VW7LBCC4N5EVQ, seg_pos={0 0}, upload=544917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:37848->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:37848->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37848->77.90.31.19:4844: use of closed network connection
Dec 31 13:20:07.488039884 (621h51m7.359122929s) [noise=true , address=77.90.31.19:4844 , piece_id=3EM5UKP234FHLWQKD2VXM3W57DMNVN3DPZGVOL2VW7LBCC4N5EVQ, seg_pos={0 0}, upload=544917]: Long tail cancellation due to optimal threshold
Dec 31 13:20:07.488259471 (621h51m7.359342517s) [noise=true , address=145.79.227.43:4158 , piece_id=LRX67YOW6E7N6NOAMMCVPQDST3UMQKXSJNYR33PF7QYS462MSHJQ, seg_pos={0 0}, upload=544917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 31 13:20:07.488281163 (621h51m7.359364208s) [noise=true , address=145.79.227.43:4158 , piece_id=LRX67YOW6E7N6NOAMMCVPQDST3UMQKXSJNYR33PF7QYS462MSHJQ, seg_pos={0 0}, upload=544917]: Long tail cancellation due to optimal threshold
Dec 31 13:20:07.488317883 (621h51m7.359400918s) [seg_pos={0 0}, upload=544917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:20:07.488459302 (621h51m7.359542347s) [upload=544917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:20:07.488466656 (621h51m7.359549701s) [upload=544917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:20:07.772371950 (621h51m7.643454985s) [upload=544917]: Done uploading segments...
Dec 31 13:20:07.772392689 (621h51m7.643475724s) [upload=544917]: Upload finished. err: <nil>
Duration: 30.003615076s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.274939368s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:20:33 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.274964045s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
54.829679875s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:20:28 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 54.829698029s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [......X.........................................XX.X...XXX..] duration:
5.001168092s trace:
https://jaeger.dev.storj.io/trace/76e318e1b1f8473a
FAIL (2025-12-31 13:20:24 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001192559s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................................X.....X...X.X.....X........] duration:
10.96402929s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:20:06 UTC)
orbiter warning: context deadline exceeded
Duration: 10.964055851s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X......X.X...X.X........XX..X.X..X....XX..XX...X..X.XX.....] duration:
5.700411213s trace:
https://jaeger.dev.storj.io/trace/4baeeb076d807c13
FAIL (2025-12-31 13:19:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:15:42.768224072 (621h46m42.639307127s) [upload=544856]: Starting upload
Dec 31 13:15:42.768319304 (621h46m42.639402359s) [upload=544856]: Done starting upload
Dec 31 13:15:42.768404616 (621h46m42.639487651s) [upload=544856]: Uploading segments...
Dec 31 13:15:42.768617902 (621h46m42.639700957s) [upload=544856]: Got next segment. Inline: false
Dec 31 13:15:42.768654983 (621h46m42.639738029s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:15:42.768661526 (621h46m42.639744561s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:15:43.015033731 (621h46m42.886116766s) [seg_pos={0 0}, upload=544856]: Begin upload segment...
Dec 31 13:15:43.015081862 (621h46m42.886164907s) [seg_pos={0 0}, upload=544856]: Done begin upload segment.
Dec 31 13:15:43.015100147 (621h46m42.886183192s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015109314 (621h46m42.886192359s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015162356 (621h46m42.886245611s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015543051 (621h46m42.886626096s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015624435 (621h46m42.88670747s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016108898 (621h46m42.887191933s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016386166 (621h46m42.887469211s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016692740 (621h46m42.887775775s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016990466 (621h46m42.888073501s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017317438 (621h46m42.888400474s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017626467 (621h46m42.888709502s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018033142 (621h46m42.889116177s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018404539 (621h46m42.889487584s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018820612 (621h46m42.889903657s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019337756 (621h46m42.890420802s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019919594 (621h46m42.891002629s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:44.623410281 (621h46m44.494493326s) [upload=544856]: Next returned nil segment
Dec 31 13:15:44.623421261 (621h46m44.494504296s) [upload=544856]: Waiting for error group managing segments...
Dec 31 13:15:44.990272094 (621h46m44.861355129s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:44.998135897 (621h46m44.869218932s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006030760 (621h46m44.877113805s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006257813 (621h46m44.877340848s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.012887997 (621h46m44.883971042s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.017707013 (621h46m44.888790048s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.021434711 (621h46m44.892517746s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.207391147 (621h46m45.078474182s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.348342898 (621h46m45.219425943s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.352889214 (621h46m45.223972249s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.420476570 (621h46m45.291559605s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.426556015 (621h46m45.29763906s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694465381 (621h46m45.565548426s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694490027 (621h46m45.565573072s) [seg_pos={0 0}, upload=544856]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:15:45.694751155 (621h46m45.56583419s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 31 13:15:45.694795218 (621h46m45.565878273s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695057027 (621h46m45.566140072s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: read tcp 188.166.225.170:47782->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695077927 (621h46m45.566160982s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695187154 (621h46m45.566270189s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:49540->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695197024 (621h46m45.566280059s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695215348 (621h46m45.566298393s) [seg_pos={0 0}, upload=544856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:15:45.695293187 (621h46m45.566376232s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:15:45.695298838 (621h46m45.566381873s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:15:45.993209107 (621h46m45.864292142s) [upload=544856]: Done uploading segments...
Dec 31 13:15:45.993226050 (621h46m45.864309085s) [upload=544856]: Upload finished. err: <nil>
Duration: 5.700454306s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
6.894930325s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:19:49 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.894952186s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
43.548667783s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:19:33 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 43.54869299s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................................X.....X...X.X.....X.......] duration:
6.879623761s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:19:31 UTC)
orbiter warning: context deadline exceeded
Duration: 6.879651464s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.......X.........................................XX.X...XXX.] duration:
3.597151391s trace:
https://jaeger.dev.storj.io/trace/76e318e1b1f8473a
FAIL (2025-12-31 13:19:30 UTC)
orbiter warning: context deadline exceeded
Duration: 3.597177399s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................................X.....X...X.X.....X......] duration:
12.644581203s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:19:18 UTC)
orbiter warning: context deadline exceeded
Duration: 12.644603674s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..X......X.X...X.X........XX..X.X..X....XX..XX...X..X.XX....] duration:
22.381311788s trace:
https://jaeger.dev.storj.io/trace/4baeeb076d807c13
FAIL (2025-12-31 13:19:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:15:42.768224072 (621h46m42.639307127s) [upload=544856]: Starting upload
Dec 31 13:15:42.768319304 (621h46m42.639402359s) [upload=544856]: Done starting upload
Dec 31 13:15:42.768404616 (621h46m42.639487651s) [upload=544856]: Uploading segments...
Dec 31 13:15:42.768617902 (621h46m42.639700957s) [upload=544856]: Got next segment. Inline: false
Dec 31 13:15:42.768654983 (621h46m42.639738029s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:15:42.768661526 (621h46m42.639744561s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:15:43.015033731 (621h46m42.886116766s) [seg_pos={0 0}, upload=544856]: Begin upload segment...
Dec 31 13:15:43.015081862 (621h46m42.886164907s) [seg_pos={0 0}, upload=544856]: Done begin upload segment.
Dec 31 13:15:43.015100147 (621h46m42.886183192s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015109314 (621h46m42.886192359s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015162356 (621h46m42.886245611s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015543051 (621h46m42.886626096s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015624435 (621h46m42.88670747s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016108898 (621h46m42.887191933s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016386166 (621h46m42.887469211s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016692740 (621h46m42.887775775s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016990466 (621h46m42.888073501s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017317438 (621h46m42.888400474s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017626467 (621h46m42.888709502s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018033142 (621h46m42.889116177s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018404539 (621h46m42.889487584s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018820612 (621h46m42.889903657s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019337756 (621h46m42.890420802s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019919594 (621h46m42.891002629s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:44.623410281 (621h46m44.494493326s) [upload=544856]: Next returned nil segment
Dec 31 13:15:44.623421261 (621h46m44.494504296s) [upload=544856]: Waiting for error group managing segments...
Dec 31 13:15:44.990272094 (621h46m44.861355129s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:44.998135897 (621h46m44.869218932s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006030760 (621h46m44.877113805s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006257813 (621h46m44.877340848s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.012887997 (621h46m44.883971042s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.017707013 (621h46m44.888790048s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.021434711 (621h46m44.892517746s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.207391147 (621h46m45.078474182s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.348342898 (621h46m45.219425943s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.352889214 (621h46m45.223972249s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.420476570 (621h46m45.291559605s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.426556015 (621h46m45.29763906s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694465381 (621h46m45.565548426s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694490027 (621h46m45.565573072s) [seg_pos={0 0}, upload=544856]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:15:45.694751155 (621h46m45.56583419s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 31 13:15:45.694795218 (621h46m45.565878273s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695057027 (621h46m45.566140072s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: read tcp 188.166.225.170:47782->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695077927 (621h46m45.566160982s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695187154 (621h46m45.566270189s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:49540->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695197024 (621h46m45.566280059s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695215348 (621h46m45.566298393s) [seg_pos={0 0}, upload=544856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:15:45.695293187 (621h46m45.566376232s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:15:45.695298838 (621h46m45.566381873s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:15:45.993209107 (621h46m45.864292142s) [upload=544856]: Done uploading segments...
Dec 31 13:15:45.993226050 (621h46m45.864309085s) [upload=544856]: Upload finished. err: <nil>
Duration: 22.381337107s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [........X.........................................XX.X...XXX] duration:
5.001300336s trace:
https://jaeger.dev.storj.io/trace/76e318e1b1f8473a
FAIL (2025-12-31 13:19:00 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001319724s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................................X.....X...X.X.....X.....] duration:
6.748958321s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:18:54 UTC)
orbiter warning: context deadline exceeded
Duration: 6.748984621s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
2.613314393s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:18:43 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 2.613339653s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................................X.....X...X.X.....X....] duration:
24.039083635s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:18:39 UTC)
orbiter warning: context deadline exceeded
Duration: 24.039111928s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [.........X.........................................XX.X...XX] duration:
5.001323457s trace:
https://jaeger.dev.storj.io/trace/2a05f983513e48e2
FAIL (2025-12-31 13:18:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001344206s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
2.738239897s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:18:00 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 2.738265296s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
22.159167406s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:17:57 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 22.159196452s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [...X......X.X...X.X........XX..X.X..X....XX..XX...X..X.XX...] duration:
6.108736325s trace:
https://jaeger.dev.storj.io/trace/4baeeb076d807c13
FAIL (2025-12-31 13:17:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:15:42.768224072 (621h46m42.639307127s) [upload=544856]: Starting upload
Dec 31 13:15:42.768319304 (621h46m42.639402359s) [upload=544856]: Done starting upload
Dec 31 13:15:42.768404616 (621h46m42.639487651s) [upload=544856]: Uploading segments...
Dec 31 13:15:42.768617902 (621h46m42.639700957s) [upload=544856]: Got next segment. Inline: false
Dec 31 13:15:42.768654983 (621h46m42.639738029s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:15:42.768661526 (621h46m42.639744561s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:15:43.015033731 (621h46m42.886116766s) [seg_pos={0 0}, upload=544856]: Begin upload segment...
Dec 31 13:15:43.015081862 (621h46m42.886164907s) [seg_pos={0 0}, upload=544856]: Done begin upload segment.
Dec 31 13:15:43.015100147 (621h46m42.886183192s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015109314 (621h46m42.886192359s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015162356 (621h46m42.886245611s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015543051 (621h46m42.886626096s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015624435 (621h46m42.88670747s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016108898 (621h46m42.887191933s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016386166 (621h46m42.887469211s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016692740 (621h46m42.887775775s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016990466 (621h46m42.888073501s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017317438 (621h46m42.888400474s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017626467 (621h46m42.888709502s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018033142 (621h46m42.889116177s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018404539 (621h46m42.889487584s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018820612 (621h46m42.889903657s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019337756 (621h46m42.890420802s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019919594 (621h46m42.891002629s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:44.623410281 (621h46m44.494493326s) [upload=544856]: Next returned nil segment
Dec 31 13:15:44.623421261 (621h46m44.494504296s) [upload=544856]: Waiting for error group managing segments...
Dec 31 13:15:44.990272094 (621h46m44.861355129s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:44.998135897 (621h46m44.869218932s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006030760 (621h46m44.877113805s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006257813 (621h46m44.877340848s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.012887997 (621h46m44.883971042s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.017707013 (621h46m44.888790048s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.021434711 (621h46m44.892517746s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.207391147 (621h46m45.078474182s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.348342898 (621h46m45.219425943s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.352889214 (621h46m45.223972249s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.420476570 (621h46m45.291559605s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.426556015 (621h46m45.29763906s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694465381 (621h46m45.565548426s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694490027 (621h46m45.565573072s) [seg_pos={0 0}, upload=544856]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:15:45.694751155 (621h46m45.56583419s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 31 13:15:45.694795218 (621h46m45.565878273s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695057027 (621h46m45.566140072s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: read tcp 188.166.225.170:47782->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695077927 (621h46m45.566160982s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695187154 (621h46m45.566270189s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:49540->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695197024 (621h46m45.566280059s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695215348 (621h46m45.566298393s) [seg_pos={0 0}, upload=544856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:15:45.695293187 (621h46m45.566376232s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:15:45.695298838 (621h46m45.566381873s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:15:45.993209107 (621h46m45.864292142s) [upload=544856]: Done uploading segments...
Dec 31 13:15:45.993226050 (621h46m45.864309085s) [upload=544856]: Upload finished. err: <nil>
Duration: 6.108761183s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.714197325s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:17:29 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.714221851s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..........X.........................................XX.X...X] duration:
5.001184768s trace:
https://jaeger.dev.storj.io/trace/f844830185448bf
FAIL (2025-12-31 13:17:22 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00120609s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................................X.....X...X.X.....X...] duration:
13.803713967s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:17:18 UTC)
orbiter warning: context deadline exceeded
Duration: 13.803743002s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [....X......X.X...X.X........XX..X.X..X....XX..XX...X..X.XX..] duration:
5.690154949s trace:
https://jaeger.dev.storj.io/trace/4baeeb076d807c13
FAIL (2025-12-31 13:16:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:15:42.768224072 (621h46m42.639307127s) [upload=544856]: Starting upload
Dec 31 13:15:42.768319304 (621h46m42.639402359s) [upload=544856]: Done starting upload
Dec 31 13:15:42.768404616 (621h46m42.639487651s) [upload=544856]: Uploading segments...
Dec 31 13:15:42.768617902 (621h46m42.639700957s) [upload=544856]: Got next segment. Inline: false
Dec 31 13:15:42.768654983 (621h46m42.639738029s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:15:42.768661526 (621h46m42.639744561s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:15:43.015033731 (621h46m42.886116766s) [seg_pos={0 0}, upload=544856]: Begin upload segment...
Dec 31 13:15:43.015081862 (621h46m42.886164907s) [seg_pos={0 0}, upload=544856]: Done begin upload segment.
Dec 31 13:15:43.015100147 (621h46m42.886183192s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015109314 (621h46m42.886192359s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015162356 (621h46m42.886245611s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015543051 (621h46m42.886626096s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015624435 (621h46m42.88670747s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016108898 (621h46m42.887191933s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016386166 (621h46m42.887469211s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016692740 (621h46m42.887775775s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016990466 (621h46m42.888073501s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017317438 (621h46m42.888400474s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017626467 (621h46m42.888709502s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018033142 (621h46m42.889116177s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018404539 (621h46m42.889487584s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018820612 (621h46m42.889903657s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019337756 (621h46m42.890420802s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019919594 (621h46m42.891002629s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:44.623410281 (621h46m44.494493326s) [upload=544856]: Next returned nil segment
Dec 31 13:15:44.623421261 (621h46m44.494504296s) [upload=544856]: Waiting for error group managing segments...
Dec 31 13:15:44.990272094 (621h46m44.861355129s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:44.998135897 (621h46m44.869218932s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006030760 (621h46m44.877113805s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006257813 (621h46m44.877340848s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.012887997 (621h46m44.883971042s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.017707013 (621h46m44.888790048s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.021434711 (621h46m44.892517746s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.207391147 (621h46m45.078474182s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.348342898 (621h46m45.219425943s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.352889214 (621h46m45.223972249s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.420476570 (621h46m45.291559605s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.426556015 (621h46m45.29763906s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694465381 (621h46m45.565548426s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694490027 (621h46m45.565573072s) [seg_pos={0 0}, upload=544856]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:15:45.694751155 (621h46m45.56583419s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 31 13:15:45.694795218 (621h46m45.565878273s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695057027 (621h46m45.566140072s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: read tcp 188.166.225.170:47782->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695077927 (621h46m45.566160982s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695187154 (621h46m45.566270189s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:49540->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695197024 (621h46m45.566280059s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695215348 (621h46m45.566298393s) [seg_pos={0 0}, upload=544856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:15:45.695293187 (621h46m45.566376232s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:15:45.695298838 (621h46m45.566381873s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:15:45.993209107 (621h46m45.864292142s) [upload=544856]: Done uploading segments...
Dec 31 13:15:45.993226050 (621h46m45.864309085s) [upload=544856]: Upload finished. err: <nil>
Duration: 5.690183784s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
27.193916092s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:16:56 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 27.193934847s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.....X......X.X...X.X........XX..X.X..X....XX..XX...X..X.XX.] duration:
6.019478958s trace:
https://jaeger.dev.storj.io/trace/4baeeb076d807c13
FAIL (2025-12-31 13:16:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:15:42.768224072 (621h46m42.639307127s) [upload=544856]: Starting upload
Dec 31 13:15:42.768319304 (621h46m42.639402359s) [upload=544856]: Done starting upload
Dec 31 13:15:42.768404616 (621h46m42.639487651s) [upload=544856]: Uploading segments...
Dec 31 13:15:42.768617902 (621h46m42.639700957s) [upload=544856]: Got next segment. Inline: false
Dec 31 13:15:42.768654983 (621h46m42.639738029s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:15:42.768661526 (621h46m42.639744561s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:15:43.015033731 (621h46m42.886116766s) [seg_pos={0 0}, upload=544856]: Begin upload segment...
Dec 31 13:15:43.015081862 (621h46m42.886164907s) [seg_pos={0 0}, upload=544856]: Done begin upload segment.
Dec 31 13:15:43.015100147 (621h46m42.886183192s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015109314 (621h46m42.886192359s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015162356 (621h46m42.886245611s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015543051 (621h46m42.886626096s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015624435 (621h46m42.88670747s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016108898 (621h46m42.887191933s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016386166 (621h46m42.887469211s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016692740 (621h46m42.887775775s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016990466 (621h46m42.888073501s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017317438 (621h46m42.888400474s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017626467 (621h46m42.888709502s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018033142 (621h46m42.889116177s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018404539 (621h46m42.889487584s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018820612 (621h46m42.889903657s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019337756 (621h46m42.890420802s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019919594 (621h46m42.891002629s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:44.623410281 (621h46m44.494493326s) [upload=544856]: Next returned nil segment
Dec 31 13:15:44.623421261 (621h46m44.494504296s) [upload=544856]: Waiting for error group managing segments...
Dec 31 13:15:44.990272094 (621h46m44.861355129s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:44.998135897 (621h46m44.869218932s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006030760 (621h46m44.877113805s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006257813 (621h46m44.877340848s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.012887997 (621h46m44.883971042s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.017707013 (621h46m44.888790048s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.021434711 (621h46m44.892517746s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.207391147 (621h46m45.078474182s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.348342898 (621h46m45.219425943s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.352889214 (621h46m45.223972249s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.420476570 (621h46m45.291559605s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.426556015 (621h46m45.29763906s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694465381 (621h46m45.565548426s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694490027 (621h46m45.565573072s) [seg_pos={0 0}, upload=544856]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:15:45.694751155 (621h46m45.56583419s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 31 13:15:45.694795218 (621h46m45.565878273s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695057027 (621h46m45.566140072s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: read tcp 188.166.225.170:47782->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695077927 (621h46m45.566160982s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695187154 (621h46m45.566270189s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:49540->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695197024 (621h46m45.566280059s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695215348 (621h46m45.566298393s) [seg_pos={0 0}, upload=544856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:15:45.695293187 (621h46m45.566376232s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:15:45.695298838 (621h46m45.566381873s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:15:45.993209107 (621h46m45.864292142s) [upload=544856]: Done uploading segments...
Dec 31 13:15:45.993226050 (621h46m45.864309085s) [upload=544856]: Upload finished. err: <nil>
Duration: 6.0195069s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
34.398651755s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:16:28 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 34.398676783s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
6.129855037s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:16:21 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.129882019s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...........X.........................................XX.X...] duration:
3.423978502s trace:
https://jaeger.dev.storj.io/trace/5659d0a113490283
FAIL (2025-12-31 13:16:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.424004633s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................................X.....X...X.X.....X..] duration:
25.823813609s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:16:15 UTC)
orbiter warning: context deadline exceeded
Duration: 25.82382973s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [......X......X.X...X.X........XX..X.X..X....XX..XX...X..X.XX] duration:
30.002680857s trace:
https://jaeger.dev.storj.io/trace/4baeeb076d807c13
FAIL (2025-12-31 13:16:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:15:42.768224072 (621h46m42.639307127s) [upload=544856]: Starting upload
Dec 31 13:15:42.768319304 (621h46m42.639402359s) [upload=544856]: Done starting upload
Dec 31 13:15:42.768404616 (621h46m42.639487651s) [upload=544856]: Uploading segments...
Dec 31 13:15:42.768617902 (621h46m42.639700957s) [upload=544856]: Got next segment. Inline: false
Dec 31 13:15:42.768654983 (621h46m42.639738029s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:15:42.768661526 (621h46m42.639744561s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:15:43.015033731 (621h46m42.886116766s) [seg_pos={0 0}, upload=544856]: Begin upload segment...
Dec 31 13:15:43.015081862 (621h46m42.886164907s) [seg_pos={0 0}, upload=544856]: Done begin upload segment.
Dec 31 13:15:43.015100147 (621h46m42.886183192s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015109314 (621h46m42.886192359s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015162356 (621h46m42.886245611s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015543051 (621h46m42.886626096s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.015624435 (621h46m42.88670747s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016108898 (621h46m42.887191933s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016386166 (621h46m42.887469211s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016692740 (621h46m42.887775775s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.016990466 (621h46m42.888073501s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017317438 (621h46m42.888400474s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.017626467 (621h46m42.888709502s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018033142 (621h46m42.889116177s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018404539 (621h46m42.889487584s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.018820612 (621h46m42.889903657s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019337756 (621h46m42.890420802s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:43.019919594 (621h46m42.891002629s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Uploading piece...
Dec 31 13:15:44.623410281 (621h46m44.494493326s) [upload=544856]: Next returned nil segment
Dec 31 13:15:44.623421261 (621h46m44.494504296s) [upload=544856]: Waiting for error group managing segments...
Dec 31 13:15:44.990272094 (621h46m44.861355129s) [noise=true , address=145.79.227.2:4114 , piece_id=DLUWBHNFBM52XZFP4AKQC5AQJMK4W27WDOUEJUGXQ552UECPJM3A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:44.998135897 (621h46m44.869218932s) [noise=true , address=145.79.227.24:4583 , piece_id=7PRQZT6EB2JY6232XZRR26TAHDD7SPJWWMUNOBRQWMUXSUIEJDJA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006030760 (621h46m44.877113805s) [noise=true , address=145.79.227.16:4910 , piece_id=QMSHSEDA4KFFS257BJSWEMA7UXIPYI2OKVVH27JNMPHLSJNBX37A, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.006257813 (621h46m44.877340848s) [noise=true , address=145.79.227.27:4690 , piece_id=TGXSDQP5E6443XANX4OUIN3FBWBQ7LNOC3ECBNRAZBTIXYBBCO6Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.012887997 (621h46m44.883971042s) [noise=true , address=145.79.227.23:4546 , piece_id=Q4GG3C2MI3UC6KHDAH7HEQULFYQYSY34BOUTREKNWTTLV6XM57LQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.017707013 (621h46m44.888790048s) [noise=true , address=145.79.227.4:4185 , piece_id=APFHOCNF3BCK5VKJ2GCFBHL4ILKSZDGCNRAENA3GIFJ5GMFQIKDA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.021434711 (621h46m44.892517746s) [noise=true , address=145.79.227.32:4835 , piece_id=P54AESHUYGKMTR4HZO5ULBDPUPZAPQVPDI4LG32IEAQLSD6ONDCA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.207391147 (621h46m45.078474182s) [noise=true , address=77.90.31.18:4805 , piece_id=3QPBNABWY6TCONR352AKH4JHRS2S57CF5I4UI6MOV5Q4PC2IRR4Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.348342898 (621h46m45.219425943s) [noise=true , address=45.91.116.5:4297 , piece_id=5C77VOXJUFVYSQBLG2V6R5VFGT2RAAH3H2AD6BF4SB3EMBK3HV3Q, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.352889214 (621h46m45.223972249s) [noise=true , address=45.91.116.13:4154 , piece_id=HK4HLCG3TTR7MDSD46RV6XEX2ZSQTOZA3E3PTEHKXTSC7FYTDZMQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.420476570 (621h46m45.291559605s) [noise=true , address=45.91.116.6:4973 , piece_id=M3LKBO77LA524EQNFI4OPKYEZL4K3Y2OFFXNBCQDFDIAWFUZMJJQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.426556015 (621h46m45.29763906s) [noise=true , address=45.91.116.10:4083 , piece_id=GNK4BOLFCZE4MWPGY2LRU6DLSJJB6LNEXULNL67NPVZUQ7VRPXPQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694465381 (621h46m45.565548426s) [noise=true , address=45.91.116.7:4010 , piece_id=6YOX3ZWH7C745425U5ZXOE3SDKGFDDAIBA5INH7IZBE5WGP5EPTA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: <nil>
Dec 31 13:15:45.694490027 (621h46m45.565573072s) [seg_pos={0 0}, upload=544856]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:15:45.694751155 (621h46m45.56583419s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF
Dec 31 13:15:45.694795218 (621h46m45.565878273s) [noise=true , address=77.90.31.11:4556 , piece_id=SMLS4UM2KCKDXBMOSAFIQMO7VEGJXWMJZXKL6D7YVVJ56DKMIHNQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695057027 (621h46m45.566140072s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: read tcp 188.166.225.170:47782->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695077927 (621h46m45.566160982s) [noise=true , address=77.90.31.19:4844 , piece_id=HCGNTW6FFRORQ357PKKVZ3YMGGKAMXQIFEVPBWKK6XIA4Z3WLUQQ, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695187154 (621h46m45.566270189s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:49540->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:45.695197024 (621h46m45.566280059s) [noise=true , address=45.91.116.12:4226 , piece_id=B44VURVKFQMOQN5IU46DZHI6ALKUXLDODTK44UIUKN6F4ZVEKFQA, seg_pos={0 0}, upload=544856]: Long tail cancellation due to optimal threshold
Dec 31 13:15:45.695215348 (621h46m45.566298393s) [seg_pos={0 0}, upload=544856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:15:45.695293187 (621h46m45.566376232s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:15:45.695298838 (621h46m45.566381873s) [upload=544856]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:15:45.993209107 (621h46m45.864292142s) [upload=544856]: Done uploading segments...
Dec 31 13:15:45.993226050 (621h46m45.864309085s) [upload=544856]: Upload finished. err: <nil>
Duration: 30.00270379s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............X.........................................XX.X..] duration:
3.72821868s trace:
https://jaeger.dev.storj.io/trace/5659d0a113490283
FAIL (2025-12-31 13:16:00 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.728246642s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
29.680115303s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:15:50 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 29.680131053s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
6.909492312s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:15:41 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.909514634s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.......X......X.X...X.X........XX..X.X..X....XX..XX...X..X.X] duration:
30.002797869s trace:
https://jaeger.dev.storj.io/trace/690b7dda71eab35d
FAIL (2025-12-31 13:15:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: RCAFUQWNHNER2ZMZQRBYBEUVY7QLR6PBUSR2EH2AODNHMM47EQVQ) context deadline exceeded; piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: OAZ4SAQNNCZNVOSHWKTUWIVGNXCI4J3DC6CFPNOXY46P5VGPWRYA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: JQP6IHLWMSSV4F3YSLJHLFE22UCJPXXNOTV7HA4CB7XKVW3SI5HA) context deadline exceeded
======LOGS======
Dec 31 13:15:04.713894362 (621h46m4.584977397s) [upload=544849]: Starting upload
Dec 31 13:15:04.713990325 (621h46m4.5850734s) [upload=544849]: Done starting upload
Dec 31 13:15:04.714244298 (621h46m4.585327343s) [upload=544849]: Uploading segments...
Dec 31 13:15:04.714298361 (621h46m4.585381406s) [upload=544849]: Got next segment. Inline: false
Dec 31 13:15:04.714337786 (621h46m4.585420831s) [upload=544849]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:15:04.714344278 (621h46m4.585427324s) [upload=544849]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:15:04.974400582 (621h46m4.845483627s) [seg_pos={0 0}, upload=544849]: Begin upload segment...
Dec 31 13:15:04.974461709 (621h46m4.845544754s) [seg_pos={0 0}, upload=544849]: Done begin upload segment.
Dec 31 13:15:04.974503007 (621h46m4.845586052s) [noise=true , address=145.79.227.19:4078 , piece_id=ISXNMEEWPQGTPL75YYSMQNGOJCGKZW7YLKPETHV6GJ74JYTJS7UA, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.974594331 (621h46m4.845677376s) [noise=true , address=45.91.116.4:4939 , piece_id=OX25PZAPIBVMPYVW5EKWDK5W3HYNMNSTK5AIKBG7KGSOI47FHUOQ, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.975199543 (621h46m4.846282588s) [noise=true , address=145.79.227.7:4293 , piece_id=CLZSUOHO7WTOSBKDONCIO37V2DEY2UARBUHQTUVYARBH2JN5QQRQ, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.975578405 (621h46m4.84666145s) [noise=true , address=145.79.227.36:4519 , piece_id=TTP7NFNV6FPADNUMJUQUHVVH3U4YKTTE4IYNQ3A4XRHYTIPF3BUA, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.976106120 (621h46m4.847189165s) [noise=true , address=45.91.116.10:4084 , piece_id=KY4ROWUI5KZJNVI5L56YEDP7XGV2AO6335Z47WNXF7OQWCXLVNMQ, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.976461227 (621h46m4.847544262s) [noise=true , address=145.79.227.14:4837 , piece_id=56CFYVEUPMCREX56PZ3QG7MDR3WAK6G6XVT6GRBDRQRVNPUJTPFQ, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.976882028 (621h46m4.847965073s) [noise=true , address=145.79.227.42:4122 , piece_id=TUL37HKGUK5X25CNUDBFDDJ3XEQMZXENTJRN2YQUMZAXCLKX472Q, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.977293752 (621h46m4.848376787s) [noise=true , address=145.79.227.51:4410 , piece_id=PS3QT7T7JPIKKWHQZA7NXAXNYGAVNUNSFC26DG3QLVVWPENEFSLQ, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.977764979 (621h46m4.848848014s) [noise=true , address=45.91.116.13:4155 , piece_id=RCAFUQWNHNER2ZMZQRBYBEUVY7QLR6PBUSR2EH2AODNHMM47EQVQ, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.978454432 (621h46m4.849537477s) [noise=true , address=77.90.31.14:4662 , piece_id=OAZ4SAQNNCZNVOSHWKTUWIVGNXCI4J3DC6CFPNOXY46P5VGPWRYA, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.979206123 (621h46m4.850289168s) [noise=true , address=77.90.31.10:4519 , piece_id=O6WZ5FO67TYYAM4LEE3Q7U3BW42XDMVEBYROCHO64E6WSJLMSBGQ, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.979881980 (621h46m4.850965025s) [noise=true , address=145.79.227.18:4007 , piece_id=WWKOFNQHB7OPVJMS5IHQX4FTCSRXAXZVJVE5EJJG6P2HSRXX5ERA, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.980513202 (621h46m4.851596237s) [noise=true , address=45.91.116.6:4975 , piece_id=IBOVLHSFFKSV55DNBJEMTKSCHFI5QS3ZE77VXRVOPC3J4MRF5MOQ, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.981268340 (621h46m4.852351385s) [noise=true , address=145.79.227.16:4910 , piece_id=ZFBDFXR32MHZFHPWJSOOC4GSE6UMSM2NYDGY2XCZTSGJAEWXGQLQ, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.981472419 (621h46m4.852555464s) [noise=true , address=77.90.31.18:4805 , piece_id=JQP6IHLWMSSV4F3YSLJHLFE22UCJPXXNOTV7HA4CB7XKVW3SI5HA, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:04.982147003 (621h46m4.853230048s) [noise=true , address=145.79.227.11:4731 , piece_id=JNKH62HWFJTAQ3E3UT6FMJDGNIOT3VMGAFGCCFHTBFFWJLWAUHCA, seg_pos={0 0}, upload=544849]: Uploading piece...
Dec 31 13:15:06.677386473 (621h46m6.548469518s) [upload=544849]: Next returned nil segment
Dec 31 13:15:06.677442489 (621h46m6.548525544s) [upload=544849]: Waiting for error group managing segments...
Dec 31 13:15:06.958629340 (621h46m6.829712385s) [noise=true , address=145.79.227.51:4410 , piece_id=PS3QT7T7JPIKKWHQZA7NXAXNYGAVNUNSFC26DG3QLVVWPENEFSLQ, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:06.960383840 (621h46m6.831466875s) [noise=true , address=145.79.227.7:4293 , piece_id=CLZSUOHO7WTOSBKDONCIO37V2DEY2UARBUHQTUVYARBH2JN5QQRQ, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:06.964201159 (621h46m6.835284214s) [noise=true , address=145.79.227.18:4007 , piece_id=WWKOFNQHB7OPVJMS5IHQX4FTCSRXAXZVJVE5EJJG6P2HSRXX5ERA, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:06.969157647 (621h46m6.840240702s) [noise=true , address=145.79.227.36:4519 , piece_id=TTP7NFNV6FPADNUMJUQUHVVH3U4YKTTE4IYNQ3A4XRHYTIPF3BUA, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:06.977233695 (621h46m6.84831674s) [noise=true , address=145.79.227.14:4837 , piece_id=56CFYVEUPMCREX56PZ3QG7MDR3WAK6G6XVT6GRBDRQRVNPUJTPFQ, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:06.982019828 (621h46m6.853102873s) [noise=true , address=145.79.227.16:4910 , piece_id=ZFBDFXR32MHZFHPWJSOOC4GSE6UMSM2NYDGY2XCZTSGJAEWXGQLQ, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:06.985859900 (621h46m6.856942945s) [noise=true , address=145.79.227.19:4078 , piece_id=ISXNMEEWPQGTPL75YYSMQNGOJCGKZW7YLKPETHV6GJ74JYTJS7UA, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:06.993025134 (621h46m6.864108189s) [noise=true , address=145.79.227.42:4122 , piece_id=TUL37HKGUK5X25CNUDBFDDJ3XEQMZXENTJRN2YQUMZAXCLKX472Q, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:07.056510780 (621h46m6.927593815s) [noise=true , address=145.79.227.11:4731 , piece_id=JNKH62HWFJTAQ3E3UT6FMJDGNIOT3VMGAFGCCFHTBFFWJLWAUHCA, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:07.140550554 (621h46m7.011633599s) [noise=true , address=77.90.31.14:4662 , piece_id=OAZ4SAQNNCZNVOSHWKTUWIVGNXCI4J3DC6CFPNOXY46P5VGPWRYA, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:07.239794699 (621h46m7.110877754s) [noise=true , address=77.90.31.18:4805 , piece_id=JQP6IHLWMSSV4F3YSLJHLFE22UCJPXXNOTV7HA4CB7XKVW3SI5HA, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:07.245046229 (621h46m7.116129284s) [noise=true , address=77.90.31.10:4519 , piece_id=O6WZ5FO67TYYAM4LEE3Q7U3BW42XDMVEBYROCHO64E6WSJLMSBGQ, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:07.368480856 (621h46m7.239563901s) [noise=true , address=45.91.116.13:4155 , piece_id=RCAFUQWNHNER2ZMZQRBYBEUVY7QLR6PBUSR2EH2AODNHMM47EQVQ, seg_pos={0 0}, upload=544849]: Done uploading piece. err: <nil>
Dec 31 13:15:07.368507085 (621h46m7.23959013s) [seg_pos={0 0}, upload=544849]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:15:07.368736993 (621h46m7.239820039s) [noise=true , address=45.91.116.4:4939 , piece_id=OX25PZAPIBVMPYVW5EKWDK5W3HYNMNSTK5AIKBG7KGSOI47FHUOQ, seg_pos={0 0}, upload=544849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 31 13:15:07.368746371 (621h46m7.239829416s) [noise=true , address=45.91.116.4:4939 , piece_id=OX25PZAPIBVMPYVW5EKWDK5W3HYNMNSTK5AIKBG7KGSOI47FHUOQ, seg_pos={0 0}, upload=544849]: Long tail cancellation due to optimal threshold
Dec 31 13:15:07.369192160 (621h46m7.240275195s) [noise=true , address=45.91.116.10:4084 , piece_id=KY4ROWUI5KZJNVI5L56YEDP7XGV2AO6335Z47WNXF7OQWCXLVNMQ, seg_pos={0 0}, upload=544849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: read tcp 188.166.225.170:43960->45.91.116.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:15:07.369201648 (621h46m7.240284693s) [noise=true , address=45.91.116.10:4084 , piece_id=KY4ROWUI5KZJNVI5L56YEDP7XGV2AO6335Z47WNXF7OQWCXLVNMQ, seg_pos={0 0}, upload=544849]: Long tail cancellation due to optimal threshold
Dec 31 13:15:07.369359659 (621h46m7.240442704s) [noise=true , address=45.91.116.6:4975 , piece_id=IBOVLHSFFKSV55DNBJEMTKSCHFI5QS3ZE77VXRVOPC3J4MRF5MOQ, seg_pos={0 0}, upload=544849]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 31 13:15:07.369367975 (621h46m7.24045102s) [noise=true , address=45.91.116.6:4975 , piece_id=IBOVLHSFFKSV55DNBJEMTKSCHFI5QS3ZE77VXRVOPC3J4MRF5MOQ, seg_pos={0 0}, upload=544849]: Long tail cancellation due to optimal threshold
Dec 31 13:15:07.369390759 (621h46m7.240473804s) [seg_pos={0 0}, upload=544849]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:15:07.369481511 (621h46m7.240564556s) [upload=544849]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:15:07.369488405 (621h46m7.24057145s) [upload=544849]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:15:07.656340351 (621h46m7.527423396s) [upload=544849]: Done uploading segments...
Dec 31 13:15:07.656365128 (621h46m7.527448174s) [upload=544849]: Upload finished. err: <nil>
Duration: 30.002828558s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
29.306760013s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:15:20 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 29.306786855s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........................................X.....X...X.X.....X.] duration:
20.614937022s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:14:59 UTC)
orbiter warning: context deadline exceeded
Duration: 20.614961168s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............X.........................................XX.X.] duration:
4.353177335s trace:
https://jaeger.dev.storj.io/trace/5659d0a113490283
FAIL (2025-12-31 13:14:56 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.35341614s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.527169467s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:14:54 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.527183806s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X.......X......X.X...X.X........XX..X.X..X....XX..XX...X..X.] duration:
6.736069193s trace:
https://jaeger.dev.storj.io/trace/34e24e39af6e80c9
FAIL (2025-12-31 13:14:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 2P77TRHLKSP4JFRJ4W652COQCC7ZQNH46WLVNPQY6VKGPECLIZRQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VRFL6SWL7O7I62WCTGENP3F3NVQXPVEEEK3QHWCA44QAV34ZC2FQ) context deadline exceeded
======LOGS======
ec 31 13:13:45.445088018 (621h44m45.316171053s) [upload=544830]: Starting upload
Dec 31 13:13:45.445142451 (621h44m45.316225496s) [upload=544830]: Done starting upload
Dec 31 13:13:45.445275064 (621h44m45.316358109s) [upload=544830]: Uploading segments...
Dec 31 13:13:45.445410301 (621h44m45.316493337s) [upload=544830]: Got next segment. Inline: false
Dec 31 13:13:45.445458292 (621h44m45.316541328s) [upload=544830]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:13:45.445465426 (621h44m45.316548461s) [upload=544830]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:13:45.690914136 (621h44m45.561997171s) [seg_pos={0 0}, upload=544830]: Begin upload segment...
Dec 31 13:13:45.690974020 (621h44m45.562057065s) [seg_pos={0 0}, upload=544830]: Done begin upload segment.
Dec 31 13:13:45.691053180 (621h44m45.562136225s) [noise=true , address=45.91.116.6:4976 , piece_id=2P77TRHLKSP4JFRJ4W652COQCC7ZQNH46WLVNPQY6VKGPECLIZRQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.691073219 (621h44m45.562156254s) [noise=true , address=77.90.31.11:4554 , piece_id=3FEHZGYQYV2F3KB7EYXFRRN7JHIDZL6WQX5ATJGDXKZFAPQ4HSNQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.691773643 (621h44m45.562856678s) [noise=true , address=45.91.116.12:4227 , piece_id=VRFL6SWL7O7I62WCTGENP3F3NVQXPVEEEK3QHWCA44QAV34ZC2FQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.692180567 (621h44m45.563263602s) [noise=true , address=77.90.31.19:4844 , piece_id=DBKDXG2O4QVMQMRH5EI3ITMOTV73XNTU3VFJJNRWQUW6WCNIVMRQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.692590478 (621h44m45.563673513s) [noise=true , address=77.90.31.16:4736 , piece_id=2IPWUWVQBH5JP5VGKEAJRC52ROPU2TRNYJ6NLXQ5OINKPZS6CPTQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.693035074 (621h44m45.564118109s) [noise=true , address=145.79.227.14:4838 , piece_id=MZ63ZRZQ3EGTUPCTS5HQK7LFF5YKBMAGPWEDN3HLSHLLJQ5D5HZQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.693525017 (621h44m45.564608062s) [noise=true , address=145.79.227.24:4582 , piece_id=VEQYTFZCSI6AMTL7MLU2I7462KIIPDLETJB7OCNMJTQAMJ2WEH2A, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.694796207 (621h44m45.565879262s) [noise=true , address=145.79.227.16:4911 , piece_id=CAGQRMUFKWKT7JXKXFDNKWOLI4E6T72PZH2RJHTALGWXN2QUHTHQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.694908381 (621h44m45.565991416s) [noise=true , address=145.79.227.11:4730 , piece_id=TRQEFVI7CORMZGKSAFMDMT2RJC3EMOHWFK3R7XRAJPRGDVJSOURA, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.695735235 (621h44m45.56681827s) [noise=true , address=45.91.116.14:4191 , piece_id=ZNA77MXE7UYLBDX5TQOXYXHNKUEQCMSOH4DRYKGN4O7XVLG66MGQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.696819800 (621h44m45.567902835s) [noise=true , address=45.91.116.9:4263 , piece_id=4BMVN26IDGL6TRFCFYKTSODSUSDHZF7WPNA3345UEUAPBFIPNQGQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.697673396 (621h44m45.568756431s) [noise=true , address=145.79.227.8:4330 , piece_id=KWXT5ISYYLZQGYJVF4VSZI6HCTOEBD5BG7RSWBPXPFSDFW33UI6A, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.697852297 (621h44m45.568935332s) [noise=true , address=145.79.227.51:4411 , piece_id=K4KZFGGEEWIEGNQTOR5Z6UZFS26OU3ZF2H2ZK53YZSPZ54IYP6VA, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.698995995 (621h44m45.57007904s) [noise=true , address=145.79.227.45:4230 , piece_id=3CBI4NOTXCE3B4KUEWERJQGMPFR4NV25HHMCVSOV5LVY5Y3VP5UQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.703857452 (621h44m45.574940487s) [noise=true , address=145.79.227.52:4446 , piece_id=QO2PNPUMIW3EQGERCSRCX3MO6F7BVMXP343UZI2QEMQLPGSFW3BA, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.705464943 (621h44m45.576547978s) [noise=true , address=45.91.116.4:4937 , piece_id=WVU2JX73VJYRMVCXZ6PCSNDYI3LYXJXGPITDXRYNEVAGQIYKU35Q, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:47.395268531 (621h44m47.266351586s) [upload=544830]: Next returned nil segment
Dec 31 13:13:47.395286565 (621h44m47.26636961s) [upload=544830]: Waiting for error group managing segments...
Dec 31 13:13:47.679366364 (621h44m47.550449409s) [noise=true , address=145.79.227.16:4911 , piece_id=CAGQRMUFKWKT7JXKXFDNKWOLI4E6T72PZH2RJHTALGWXN2QUHTHQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.687919660 (621h44m47.559002706s) [noise=true , address=145.79.227.45:4230 , piece_id=3CBI4NOTXCE3B4KUEWERJQGMPFR4NV25HHMCVSOV5LVY5Y3VP5UQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.694780885 (621h44m47.56586393s) [noise=true , address=145.79.227.51:4411 , piece_id=K4KZFGGEEWIEGNQTOR5Z6UZFS26OU3ZF2H2ZK53YZSPZ54IYP6VA, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.702934832 (621h44m47.574017867s) [noise=true , address=145.79.227.8:4330 , piece_id=KWXT5ISYYLZQGYJVF4VSZI6HCTOEBD5BG7RSWBPXPFSDFW33UI6A, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.710486010 (621h44m47.581569056s) [noise=true , address=145.79.227.52:4446 , piece_id=QO2PNPUMIW3EQGERCSRCX3MO6F7BVMXP343UZI2QEMQLPGSFW3BA, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.711381675 (621h44m47.58246472s) [noise=true , address=145.79.227.24:4582 , piece_id=VEQYTFZCSI6AMTL7MLU2I7462KIIPDLETJB7OCNMJTQAMJ2WEH2A, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.720669481 (621h44m47.591752516s) [noise=true , address=145.79.227.14:4838 , piece_id=MZ63ZRZQ3EGTUPCTS5HQK7LFF5YKBMAGPWEDN3HLSHLLJQ5D5HZQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.766529916 (621h44m47.637612951s) [noise=true , address=145.79.227.11:4730 , piece_id=TRQEFVI7CORMZGKSAFMDMT2RJC3EMOHWFK3R7XRAJPRGDVJSOURA, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.851090171 (621h44m47.722173216s) [noise=true , address=77.90.31.11:4554 , piece_id=3FEHZGYQYV2F3KB7EYXFRRN7JHIDZL6WQX5ATJGDXKZFAPQ4HSNQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.967294761 (621h44m47.838377816s) [noise=true , address=77.90.31.19:4844 , piece_id=DBKDXG2O4QVMQMRH5EI3ITMOTV73XNTU3VFJJNRWQUW6WCNIVMRQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:48.029939857 (621h44m47.901022892s) [noise=true , address=45.91.116.14:4191 , piece_id=ZNA77MXE7UYLBDX5TQOXYXHNKUEQCMSOH4DRYKGN4O7XVLG66MGQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:48.076199962 (621h44m47.947283017s) [noise=true , address=45.91.116.12:4227 , piece_id=VRFL6SWL7O7I62WCTGENP3F3NVQXPVEEEK3QHWCA44QAV34ZC2FQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:48.272352134 (621h44m48.143435169s) [noise=true , address=45.91.116.6:4976 , piece_id=2P77TRHLKSP4JFRJ4W652COQCC7ZQNH46WLVNPQY6VKGPECLIZRQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:48.272370519 (621h44m48.143453555s) [seg_pos={0 0}, upload=544830]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:13:48.272549650 (621h44m48.143632685s) [noise=true , address=45.91.116.9:4263 , piece_id=4BMVN26IDGL6TRFCFYKTSODSUSDHZF7WPNA3345UEUAPBFIPNQGQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:37720->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:37720->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37720->45.91.116.9:4263: use of closed network connection
Dec 31 13:13:48.272555541 (621h44m48.143638577s) [noise=true , address=45.91.116.9:4263 , piece_id=4BMVN26IDGL6TRFCFYKTSODSUSDHZF7WPNA3345UEUAPBFIPNQGQ, seg_pos={0 0}, upload=544830]: Long tail cancellation due to optimal threshold
Dec 31 13:13:48.272616257 (621h44m48.143699302s) [noise=true , address=45.91.116.4:4937 , piece_id=WVU2JX73VJYRMVCXZ6PCSNDYI3LYXJXGPITDXRYNEVAGQIYKU35Q, seg_pos={0 0}, upload=544830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 31 13:13:48.272625064 (621h44m48.143708099s) [noise=true , address=45.91.116.4:4937 , piece_id=WVU2JX73VJYRMVCXZ6PCSNDYI3LYXJXGPITDXRYNEVAGQIYKU35Q, seg_pos={0 0}, upload=544830]: Long tail cancellation due to optimal threshold
Dec 31 13:13:48.272845163 (621h44m48.143928258s) [noise=true , address=77.90.31.16:4736 , piece_id=2IPWUWVQBH5JP5VGKEAJRC52ROPU2TRNYJ6NLXQ5OINKPZS6CPTQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 13:13:48.272854370 (621h44m48.143937405s) [noise=true , address=77.90.31.16:4736 , piece_id=2IPWUWVQBH5JP5VGKEAJRC52ROPU2TRNYJ6NLXQ5OINKPZS6CPTQ, seg_pos={0 0}, upload=544830]: Long tail cancellation due to optimal threshold
Dec 31 13:13:48.272878767 (621h44m48.143961812s) [seg_pos={0 0}, upload=544830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:13:48.272975762 (621h44m48.144058807s) [upload=544830]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:13:48.272982925 (621h44m48.14406596s) [upload=544830]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:13:48.559896260 (621h44m48.430979296s) [upload=544830]: Done uploading segments...
Dec 31 13:13:48.559914976 (621h44m48.430998011s) [upload=544830]: Upload finished. err: <nil>
Duration: 6.736101936s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..............X.........................................XX.X] duration:
5.001693219s trace:
https://jaeger.dev.storj.io/trace/5659d0a113490283
FAIL (2025-12-31 13:14:35 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001712375s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........................................X.....X...X.X.....X] duration:
30.001874211s trace:
https://jaeger.dev.storj.io/trace/dee23ec4584b2e1
FAIL (2025-12-31 13:14:30 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001896145s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
6.948357933s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:14:29 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.948381569s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
45.981307104s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:14:24 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 45.981322905s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.X.......X......X.X...X.X........XX..X.X..X....XX..XX...X..X] duration:
30.003760216s trace:
https://jaeger.dev.storj.io/trace/34e24e39af6e80c9
FAIL (2025-12-31 13:14:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 2P77TRHLKSP4JFRJ4W652COQCC7ZQNH46WLVNPQY6VKGPECLIZRQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VRFL6SWL7O7I62WCTGENP3F3NVQXPVEEEK3QHWCA44QAV34ZC2FQ) context deadline exceeded
======LOGS======
ec 31 13:13:45.445088018 (621h44m45.316171053s) [upload=544830]: Starting upload
Dec 31 13:13:45.445142451 (621h44m45.316225496s) [upload=544830]: Done starting upload
Dec 31 13:13:45.445275064 (621h44m45.316358109s) [upload=544830]: Uploading segments...
Dec 31 13:13:45.445410301 (621h44m45.316493337s) [upload=544830]: Got next segment. Inline: false
Dec 31 13:13:45.445458292 (621h44m45.316541328s) [upload=544830]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:13:45.445465426 (621h44m45.316548461s) [upload=544830]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:13:45.690914136 (621h44m45.561997171s) [seg_pos={0 0}, upload=544830]: Begin upload segment...
Dec 31 13:13:45.690974020 (621h44m45.562057065s) [seg_pos={0 0}, upload=544830]: Done begin upload segment.
Dec 31 13:13:45.691053180 (621h44m45.562136225s) [noise=true , address=45.91.116.6:4976 , piece_id=2P77TRHLKSP4JFRJ4W652COQCC7ZQNH46WLVNPQY6VKGPECLIZRQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.691073219 (621h44m45.562156254s) [noise=true , address=77.90.31.11:4554 , piece_id=3FEHZGYQYV2F3KB7EYXFRRN7JHIDZL6WQX5ATJGDXKZFAPQ4HSNQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.691773643 (621h44m45.562856678s) [noise=true , address=45.91.116.12:4227 , piece_id=VRFL6SWL7O7I62WCTGENP3F3NVQXPVEEEK3QHWCA44QAV34ZC2FQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.692180567 (621h44m45.563263602s) [noise=true , address=77.90.31.19:4844 , piece_id=DBKDXG2O4QVMQMRH5EI3ITMOTV73XNTU3VFJJNRWQUW6WCNIVMRQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.692590478 (621h44m45.563673513s) [noise=true , address=77.90.31.16:4736 , piece_id=2IPWUWVQBH5JP5VGKEAJRC52ROPU2TRNYJ6NLXQ5OINKPZS6CPTQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.693035074 (621h44m45.564118109s) [noise=true , address=145.79.227.14:4838 , piece_id=MZ63ZRZQ3EGTUPCTS5HQK7LFF5YKBMAGPWEDN3HLSHLLJQ5D5HZQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.693525017 (621h44m45.564608062s) [noise=true , address=145.79.227.24:4582 , piece_id=VEQYTFZCSI6AMTL7MLU2I7462KIIPDLETJB7OCNMJTQAMJ2WEH2A, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.694796207 (621h44m45.565879262s) [noise=true , address=145.79.227.16:4911 , piece_id=CAGQRMUFKWKT7JXKXFDNKWOLI4E6T72PZH2RJHTALGWXN2QUHTHQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.694908381 (621h44m45.565991416s) [noise=true , address=145.79.227.11:4730 , piece_id=TRQEFVI7CORMZGKSAFMDMT2RJC3EMOHWFK3R7XRAJPRGDVJSOURA, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.695735235 (621h44m45.56681827s) [noise=true , address=45.91.116.14:4191 , piece_id=ZNA77MXE7UYLBDX5TQOXYXHNKUEQCMSOH4DRYKGN4O7XVLG66MGQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.696819800 (621h44m45.567902835s) [noise=true , address=45.91.116.9:4263 , piece_id=4BMVN26IDGL6TRFCFYKTSODSUSDHZF7WPNA3345UEUAPBFIPNQGQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.697673396 (621h44m45.568756431s) [noise=true , address=145.79.227.8:4330 , piece_id=KWXT5ISYYLZQGYJVF4VSZI6HCTOEBD5BG7RSWBPXPFSDFW33UI6A, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.697852297 (621h44m45.568935332s) [noise=true , address=145.79.227.51:4411 , piece_id=K4KZFGGEEWIEGNQTOR5Z6UZFS26OU3ZF2H2ZK53YZSPZ54IYP6VA, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.698995995 (621h44m45.57007904s) [noise=true , address=145.79.227.45:4230 , piece_id=3CBI4NOTXCE3B4KUEWERJQGMPFR4NV25HHMCVSOV5LVY5Y3VP5UQ, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.703857452 (621h44m45.574940487s) [noise=true , address=145.79.227.52:4446 , piece_id=QO2PNPUMIW3EQGERCSRCX3MO6F7BVMXP343UZI2QEMQLPGSFW3BA, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:45.705464943 (621h44m45.576547978s) [noise=true , address=45.91.116.4:4937 , piece_id=WVU2JX73VJYRMVCXZ6PCSNDYI3LYXJXGPITDXRYNEVAGQIYKU35Q, seg_pos={0 0}, upload=544830]: Uploading piece...
Dec 31 13:13:47.395268531 (621h44m47.266351586s) [upload=544830]: Next returned nil segment
Dec 31 13:13:47.395286565 (621h44m47.26636961s) [upload=544830]: Waiting for error group managing segments...
Dec 31 13:13:47.679366364 (621h44m47.550449409s) [noise=true , address=145.79.227.16:4911 , piece_id=CAGQRMUFKWKT7JXKXFDNKWOLI4E6T72PZH2RJHTALGWXN2QUHTHQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.687919660 (621h44m47.559002706s) [noise=true , address=145.79.227.45:4230 , piece_id=3CBI4NOTXCE3B4KUEWERJQGMPFR4NV25HHMCVSOV5LVY5Y3VP5UQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.694780885 (621h44m47.56586393s) [noise=true , address=145.79.227.51:4411 , piece_id=K4KZFGGEEWIEGNQTOR5Z6UZFS26OU3ZF2H2ZK53YZSPZ54IYP6VA, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.702934832 (621h44m47.574017867s) [noise=true , address=145.79.227.8:4330 , piece_id=KWXT5ISYYLZQGYJVF4VSZI6HCTOEBD5BG7RSWBPXPFSDFW33UI6A, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.710486010 (621h44m47.581569056s) [noise=true , address=145.79.227.52:4446 , piece_id=QO2PNPUMIW3EQGERCSRCX3MO6F7BVMXP343UZI2QEMQLPGSFW3BA, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.711381675 (621h44m47.58246472s) [noise=true , address=145.79.227.24:4582 , piece_id=VEQYTFZCSI6AMTL7MLU2I7462KIIPDLETJB7OCNMJTQAMJ2WEH2A, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.720669481 (621h44m47.591752516s) [noise=true , address=145.79.227.14:4838 , piece_id=MZ63ZRZQ3EGTUPCTS5HQK7LFF5YKBMAGPWEDN3HLSHLLJQ5D5HZQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.766529916 (621h44m47.637612951s) [noise=true , address=145.79.227.11:4730 , piece_id=TRQEFVI7CORMZGKSAFMDMT2RJC3EMOHWFK3R7XRAJPRGDVJSOURA, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.851090171 (621h44m47.722173216s) [noise=true , address=77.90.31.11:4554 , piece_id=3FEHZGYQYV2F3KB7EYXFRRN7JHIDZL6WQX5ATJGDXKZFAPQ4HSNQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:47.967294761 (621h44m47.838377816s) [noise=true , address=77.90.31.19:4844 , piece_id=DBKDXG2O4QVMQMRH5EI3ITMOTV73XNTU3VFJJNRWQUW6WCNIVMRQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:48.029939857 (621h44m47.901022892s) [noise=true , address=45.91.116.14:4191 , piece_id=ZNA77MXE7UYLBDX5TQOXYXHNKUEQCMSOH4DRYKGN4O7XVLG66MGQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:48.076199962 (621h44m47.947283017s) [noise=true , address=45.91.116.12:4227 , piece_id=VRFL6SWL7O7I62WCTGENP3F3NVQXPVEEEK3QHWCA44QAV34ZC2FQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:48.272352134 (621h44m48.143435169s) [noise=true , address=45.91.116.6:4976 , piece_id=2P77TRHLKSP4JFRJ4W652COQCC7ZQNH46WLVNPQY6VKGPECLIZRQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: <nil>
Dec 31 13:13:48.272370519 (621h44m48.143453555s) [seg_pos={0 0}, upload=544830]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:13:48.272549650 (621h44m48.143632685s) [noise=true , address=45.91.116.9:4263 , piece_id=4BMVN26IDGL6TRFCFYKTSODSUSDHZF7WPNA3345UEUAPBFIPNQGQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:37720->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:37720->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37720->45.91.116.9:4263: use of closed network connection
Dec 31 13:13:48.272555541 (621h44m48.143638577s) [noise=true , address=45.91.116.9:4263 , piece_id=4BMVN26IDGL6TRFCFYKTSODSUSDHZF7WPNA3345UEUAPBFIPNQGQ, seg_pos={0 0}, upload=544830]: Long tail cancellation due to optimal threshold
Dec 31 13:13:48.272616257 (621h44m48.143699302s) [noise=true , address=45.91.116.4:4937 , piece_id=WVU2JX73VJYRMVCXZ6PCSNDYI3LYXJXGPITDXRYNEVAGQIYKU35Q, seg_pos={0 0}, upload=544830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 31 13:13:48.272625064 (621h44m48.143708099s) [noise=true , address=45.91.116.4:4937 , piece_id=WVU2JX73VJYRMVCXZ6PCSNDYI3LYXJXGPITDXRYNEVAGQIYKU35Q, seg_pos={0 0}, upload=544830]: Long tail cancellation due to optimal threshold
Dec 31 13:13:48.272845163 (621h44m48.143928258s) [noise=true , address=77.90.31.16:4736 , piece_id=2IPWUWVQBH5JP5VGKEAJRC52ROPU2TRNYJ6NLXQ5OINKPZS6CPTQ, seg_pos={0 0}, upload=544830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 13:13:48.272854370 (621h44m48.143937405s) [noise=true , address=77.90.31.16:4736 , piece_id=2IPWUWVQBH5JP5VGKEAJRC52ROPU2TRNYJ6NLXQ5OINKPZS6CPTQ, seg_pos={0 0}, upload=544830]: Long tail cancellation due to optimal threshold
Dec 31 13:13:48.272878767 (621h44m48.143961812s) [seg_pos={0 0}, upload=544830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:13:48.272975762 (621h44m48.144058807s) [upload=544830]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:13:48.272982925 (621h44m48.14406596s) [upload=544830]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:13:48.559896260 (621h44m48.430979296s) [upload=544830]: Done uploading segments...
Dec 31 13:13:48.559914976 (621h44m48.430998011s) [upload=544830]: Upload finished. err: <nil>
Duration: 30.003858704s
core
uplink
us-select-1
|