Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
4.989822282s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:09:21 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 4.989842902s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................X......................................] duration:
3.492590776s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:09:19 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.492607808s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [......X.................X...................................] duration:
46.593151532s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:09:16 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 46.593184724s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................................X.....X...X] duration:
30.00211006s trace:
https://jaeger.dev.storj.io/trace/1b3987f4664acddd
FAIL (2025-12-31 13:09:13 UTC)
orbiter warning: context deadline exceeded
Duration: 30.002135348s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [X..X...X.......X......X.X...X.X........XX..X.X..X....XX..XX.] duration:
5.835105622s trace:
https://jaeger.dev.storj.io/trace/14eb32d172b8e805
FAIL (2025-12-31 13:08:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:07:37.580571966 (621h38m37.451655011s) [upload=544735]: Starting upload
Dec 31 13:07:37.580690922 (621h38m37.451773957s) [upload=544735]: Done starting upload
Dec 31 13:07:37.580757438 (621h38m37.451840483s) [upload=544735]: Uploading segments...
Dec 31 13:07:37.580871856 (621h38m37.451954891s) [upload=544735]: Got next segment. Inline: false
Dec 31 13:07:37.580910770 (621h38m37.451993815s) [upload=544735]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:07:37.580916331 (621h38m37.451999376s) [upload=544735]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:07:37.826170070 (621h38m37.697253115s) [seg_pos={0 0}, upload=544735]: Begin upload segment...
Dec 31 13:07:37.826232619 (621h38m37.697315664s) [seg_pos={0 0}, upload=544735]: Done begin upload segment.
Dec 31 13:07:37.826270601 (621h38m37.697353636s) [noise=true , address=45.91.116.12:4227 , piece_id=UMU3YBLMKL5TA3I4XZYD6JFFJ4AMHN2FZ3NB227QN3LP3S3KMXHA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.826288064 (621h38m37.697371109s) [noise=true , address=145.79.227.34:4905 , piece_id=DOU3H2BQNW7HKQ5VZ5MVUOHHWF6OD67HOG3ACWUGBT7IM3Q6LCHA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.826852499 (621h38m37.697935544s) [noise=true , address=145.79.227.12:4765 , piece_id=ONA3APCG53LRTZGRHHHWAAORR5SGHBHSFIVU6OH346RGL5P5PFYA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.827175354 (621h38m37.698258389s) [noise=true , address=145.79.227.41:4087 , piece_id=4NAHQPXVCOXNK4C6TB5XMEE2ODSMGZORRRSTOMQM2NO7RQ7P6RUA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.827460897 (621h38m37.698543932s) [noise=true , address=145.79.227.48:4303 , piece_id=XRHUHHDDGAD4CJNDAUN3O6ALLIT22MUWJLEHAVQKWZZ5Q6KSEGSQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.828337597 (621h38m37.699420632s) [noise=true , address=145.79.227.5:4221 , piece_id=QPEXTQN54JUWBHXNBHNL3PXLJ6ZCYQ4MTMW6667UGVMU7HZJBS6Q, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.828728671 (621h38m37.699811706s) [noise=true , address=145.79.227.23:4547 , piece_id=MHQLLBRCBITNGVJIWPKGNWFL4EUREXWOFQXWJ63677TYZCUVTHWQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.828942718 (621h38m37.700025764s) [noise=true , address=145.79.227.7:4294 , piece_id=T3FVAOKI3MKMD5ZLTXRRYW5XW4L73IVJCC26EUJHIVHAYTRK4UCQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.829174620 (621h38m37.700257665s) [noise=true , address=145.79.227.39:4013 , piece_id=MZ6WK2YSN5YHAT5VZ7D4TXCVJZEHJNBW2IE2IIJKO67H6TZ46IUQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.829625579 (621h38m37.700708624s) [noise=true , address=45.91.116.10:4082 , piece_id=6GQV72QGXNBRH53RUWSZIIGO5AA6SJZA6BSDO7AKKG25Z24EXNWQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.829679972 (621h38m37.700763017s) [noise=true , address=145.79.227.31:4798 , piece_id=R3P46Y5X4PTARLY3PSG45LYMHHX2UJ22JROCTKOE7JDVQFLK6BSQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.830004571 (621h38m37.701087616s) [noise=true , address=77.90.31.13:4626 , piece_id=2ELYH6QQQDTIRJ4PV4MPMSA2QHBMIOAFTDMGPN7M5EBTESFWZU6Q, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.830055998 (621h38m37.701139033s) [noise=true , address=145.79.227.15:4875 , piece_id=6AE5CPDQXRGPVBOHPCTJ6H7Y4T7IS6SH5FJ7E6C7RLMFLQRR24MA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.830518018 (621h38m37.701601053s) [noise=true , address=145.79.227.47:4266 , piece_id=UKSXLAGJSLXWUJKJZDU2UH766AYZRYATFPSWNKXYYWALNGERDNGA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.830887652 (621h38m37.701970697s) [noise=true , address=77.90.31.10:4520 , piece_id=5T4L2XHK42SW2VBVE2YI2XBNOI6SD2QPWTXBU6RYQYFNFFT7AAEQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.831700710 (621h38m37.702783745s) [noise=true , address=45.91.116.9:4261 , piece_id=XOPXTR3GXDMYDO22TBBKES46RZMXUYBRLB6LLZYMJMAUTHPAKHIQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:39.542108032 (621h38m39.413191067s) [upload=544735]: Next returned nil segment
Dec 31 13:07:39.542123853 (621h38m39.413206888s) [upload=544735]: Waiting for error group managing segments...
Dec 31 13:07:39.817500158 (621h38m39.688583193s) [noise=true , address=145.79.227.12:4765 , piece_id=ONA3APCG53LRTZGRHHHWAAORR5SGHBHSFIVU6OH346RGL5P5PFYA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.819730606 (621h38m39.690813651s) [noise=true , address=145.79.227.7:4294 , piece_id=T3FVAOKI3MKMD5ZLTXRRYW5XW4L73IVJCC26EUJHIVHAYTRK4UCQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.819935426 (621h38m39.691018471s) [noise=true , address=145.79.227.48:4303 , piece_id=XRHUHHDDGAD4CJNDAUN3O6ALLIT22MUWJLEHAVQKWZZ5Q6KSEGSQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.846148459 (621h38m39.717231494s) [noise=true , address=145.79.227.31:4798 , piece_id=R3P46Y5X4PTARLY3PSG45LYMHHX2UJ22JROCTKOE7JDVQFLK6BSQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.854365997 (621h38m39.725449032s) [noise=true , address=145.79.227.47:4266 , piece_id=UKSXLAGJSLXWUJKJZDU2UH766AYZRYATFPSWNKXYYWALNGERDNGA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.888015600 (621h38m39.759098635s) [noise=true , address=145.79.227.23:4547 , piece_id=MHQLLBRCBITNGVJIWPKGNWFL4EUREXWOFQXWJ63677TYZCUVTHWQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.899900451 (621h38m39.770983486s) [noise=true , address=145.79.227.41:4087 , piece_id=4NAHQPXVCOXNK4C6TB5XMEE2ODSMGZORRRSTOMQM2NO7RQ7P6RUA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.899937080 (621h38m39.771020115s) [noise=true , address=145.79.227.15:4875 , piece_id=6AE5CPDQXRGPVBOHPCTJ6H7Y4T7IS6SH5FJ7E6C7RLMFLQRR24MA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.909266034 (621h38m39.780349069s) [noise=true , address=145.79.227.39:4013 , piece_id=MZ6WK2YSN5YHAT5VZ7D4TXCVJZEHJNBW2IE2IIJKO67H6TZ46IUQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.909923637 (621h38m39.781006682s) [noise=true , address=145.79.227.5:4221 , piece_id=QPEXTQN54JUWBHXNBHNL3PXLJ6ZCYQ4MTMW6667UGVMU7HZJBS6Q, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:40.150148398 (621h38m40.021231443s) [noise=true , address=45.91.116.10:4082 , piece_id=6GQV72QGXNBRH53RUWSZIIGO5AA6SJZA6BSDO7AKKG25Z24EXNWQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:40.154041841 (621h38m40.025124876s) [noise=true , address=45.91.116.12:4227 , piece_id=UMU3YBLMKL5TA3I4XZYD6JFFJ4AMHN2FZ3NB227QN3LP3S3KMXHA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:40.168507246 (621h38m40.039590291s) [noise=true , address=45.91.116.9:4261 , piece_id=XOPXTR3GXDMYDO22TBBKES46RZMXUYBRLB6LLZYMJMAUTHPAKHIQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:40.168527746 (621h38m40.039610781s) [seg_pos={0 0}, upload=544735]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:07:40.168737756 (621h38m40.039820791s) [noise=true , address=77.90.31.13:4626 , piece_id=2ELYH6QQQDTIRJ4PV4MPMSA2QHBMIOAFTDMGPN7M5EBTESFWZU6Q, seg_pos={0 0}, upload=544735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 31 13:07:40.168744188 (621h38m40.039827223s) [noise=true , address=77.90.31.13:4626 , piece_id=2ELYH6QQQDTIRJ4PV4MPMSA2QHBMIOAFTDMGPN7M5EBTESFWZU6Q, seg_pos={0 0}, upload=544735]: Long tail cancellation due to optimal threshold
Dec 31 13:07:40.168865698 (621h38m40.039948734s) [noise=true , address=145.79.227.34:4905 , piece_id=DOU3H2BQNW7HKQ5VZ5MVUOHHWF6OD67HOG3ACWUGBT7IM3Q6LCHA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 31 13:07:40.168882441 (621h38m40.039965476s) [noise=true , address=145.79.227.34:4905 , piece_id=DOU3H2BQNW7HKQ5VZ5MVUOHHWF6OD67HOG3ACWUGBT7IM3Q6LCHA, seg_pos={0 0}, upload=544735]: Long tail cancellation due to optimal threshold
Dec 31 13:07:40.169044589 (621h38m40.040127634s) [noise=true , address=77.90.31.10:4520 , piece_id=5T4L2XHK42SW2VBVE2YI2XBNOI6SD2QPWTXBU6RYQYFNFFT7AAEQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 31 13:07:40.169053186 (621h38m40.040136231s) [noise=true , address=77.90.31.10:4520 , piece_id=5T4L2XHK42SW2VBVE2YI2XBNOI6SD2QPWTXBU6RYQYFNFFT7AAEQ, seg_pos={0 0}, upload=544735]: Long tail cancellation due to optimal threshold
Dec 31 13:07:40.169074887 (621h38m40.040157922s) [seg_pos={0 0}, upload=544735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:07:40.169147224 (621h38m40.04023026s) [upload=544735]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:07:40.169152686 (621h38m40.040235721s) [upload=544735]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:07:40.464018252 (621h38m40.335101287s) [upload=544735]: Done uploading segments...
Dec 31 13:07:40.464039242 (621h38m40.335122277s) [upload=544735]: Upload finished. err: <nil>
Duration: 5.835135448s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................X.....................................] duration:
3.584003859s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:08:52 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.584028416s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
5.322094333s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:08:52 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.322118509s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......X.................X..................................] duration:
35.721811512s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:08:27 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 35.721826821s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X.....X...] duration:
12.369355251s trace:
https://jaeger.dev.storj.io/trace/2513d132f2ff5c93
FAIL (2025-12-31 13:08:15 UTC)
orbiter warning: context deadline exceeded
Duration: 12.36938119s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......................X....................................] duration:
3.649543646s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:08:14 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.649577851s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [XX..X...X.......X......X.X...X.X........XX..X.X..X....XX..XX] duration:
30.004160095s trace:
https://jaeger.dev.storj.io/trace/14eb32d172b8e805
FAIL (2025-12-31 13:08:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:07:37.580571966 (621h38m37.451655011s) [upload=544735]: Starting upload
Dec 31 13:07:37.580690922 (621h38m37.451773957s) [upload=544735]: Done starting upload
Dec 31 13:07:37.580757438 (621h38m37.451840483s) [upload=544735]: Uploading segments...
Dec 31 13:07:37.580871856 (621h38m37.451954891s) [upload=544735]: Got next segment. Inline: false
Dec 31 13:07:37.580910770 (621h38m37.451993815s) [upload=544735]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:07:37.580916331 (621h38m37.451999376s) [upload=544735]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:07:37.826170070 (621h38m37.697253115s) [seg_pos={0 0}, upload=544735]: Begin upload segment...
Dec 31 13:07:37.826232619 (621h38m37.697315664s) [seg_pos={0 0}, upload=544735]: Done begin upload segment.
Dec 31 13:07:37.826270601 (621h38m37.697353636s) [noise=true , address=45.91.116.12:4227 , piece_id=UMU3YBLMKL5TA3I4XZYD6JFFJ4AMHN2FZ3NB227QN3LP3S3KMXHA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.826288064 (621h38m37.697371109s) [noise=true , address=145.79.227.34:4905 , piece_id=DOU3H2BQNW7HKQ5VZ5MVUOHHWF6OD67HOG3ACWUGBT7IM3Q6LCHA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.826852499 (621h38m37.697935544s) [noise=true , address=145.79.227.12:4765 , piece_id=ONA3APCG53LRTZGRHHHWAAORR5SGHBHSFIVU6OH346RGL5P5PFYA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.827175354 (621h38m37.698258389s) [noise=true , address=145.79.227.41:4087 , piece_id=4NAHQPXVCOXNK4C6TB5XMEE2ODSMGZORRRSTOMQM2NO7RQ7P6RUA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.827460897 (621h38m37.698543932s) [noise=true , address=145.79.227.48:4303 , piece_id=XRHUHHDDGAD4CJNDAUN3O6ALLIT22MUWJLEHAVQKWZZ5Q6KSEGSQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.828337597 (621h38m37.699420632s) [noise=true , address=145.79.227.5:4221 , piece_id=QPEXTQN54JUWBHXNBHNL3PXLJ6ZCYQ4MTMW6667UGVMU7HZJBS6Q, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.828728671 (621h38m37.699811706s) [noise=true , address=145.79.227.23:4547 , piece_id=MHQLLBRCBITNGVJIWPKGNWFL4EUREXWOFQXWJ63677TYZCUVTHWQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.828942718 (621h38m37.700025764s) [noise=true , address=145.79.227.7:4294 , piece_id=T3FVAOKI3MKMD5ZLTXRRYW5XW4L73IVJCC26EUJHIVHAYTRK4UCQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.829174620 (621h38m37.700257665s) [noise=true , address=145.79.227.39:4013 , piece_id=MZ6WK2YSN5YHAT5VZ7D4TXCVJZEHJNBW2IE2IIJKO67H6TZ46IUQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.829625579 (621h38m37.700708624s) [noise=true , address=45.91.116.10:4082 , piece_id=6GQV72QGXNBRH53RUWSZIIGO5AA6SJZA6BSDO7AKKG25Z24EXNWQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.829679972 (621h38m37.700763017s) [noise=true , address=145.79.227.31:4798 , piece_id=R3P46Y5X4PTARLY3PSG45LYMHHX2UJ22JROCTKOE7JDVQFLK6BSQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.830004571 (621h38m37.701087616s) [noise=true , address=77.90.31.13:4626 , piece_id=2ELYH6QQQDTIRJ4PV4MPMSA2QHBMIOAFTDMGPN7M5EBTESFWZU6Q, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.830055998 (621h38m37.701139033s) [noise=true , address=145.79.227.15:4875 , piece_id=6AE5CPDQXRGPVBOHPCTJ6H7Y4T7IS6SH5FJ7E6C7RLMFLQRR24MA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.830518018 (621h38m37.701601053s) [noise=true , address=145.79.227.47:4266 , piece_id=UKSXLAGJSLXWUJKJZDU2UH766AYZRYATFPSWNKXYYWALNGERDNGA, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.830887652 (621h38m37.701970697s) [noise=true , address=77.90.31.10:4520 , piece_id=5T4L2XHK42SW2VBVE2YI2XBNOI6SD2QPWTXBU6RYQYFNFFT7AAEQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:37.831700710 (621h38m37.702783745s) [noise=true , address=45.91.116.9:4261 , piece_id=XOPXTR3GXDMYDO22TBBKES46RZMXUYBRLB6LLZYMJMAUTHPAKHIQ, seg_pos={0 0}, upload=544735]: Uploading piece...
Dec 31 13:07:39.542108032 (621h38m39.413191067s) [upload=544735]: Next returned nil segment
Dec 31 13:07:39.542123853 (621h38m39.413206888s) [upload=544735]: Waiting for error group managing segments...
Dec 31 13:07:39.817500158 (621h38m39.688583193s) [noise=true , address=145.79.227.12:4765 , piece_id=ONA3APCG53LRTZGRHHHWAAORR5SGHBHSFIVU6OH346RGL5P5PFYA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.819730606 (621h38m39.690813651s) [noise=true , address=145.79.227.7:4294 , piece_id=T3FVAOKI3MKMD5ZLTXRRYW5XW4L73IVJCC26EUJHIVHAYTRK4UCQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.819935426 (621h38m39.691018471s) [noise=true , address=145.79.227.48:4303 , piece_id=XRHUHHDDGAD4CJNDAUN3O6ALLIT22MUWJLEHAVQKWZZ5Q6KSEGSQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.846148459 (621h38m39.717231494s) [noise=true , address=145.79.227.31:4798 , piece_id=R3P46Y5X4PTARLY3PSG45LYMHHX2UJ22JROCTKOE7JDVQFLK6BSQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.854365997 (621h38m39.725449032s) [noise=true , address=145.79.227.47:4266 , piece_id=UKSXLAGJSLXWUJKJZDU2UH766AYZRYATFPSWNKXYYWALNGERDNGA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.888015600 (621h38m39.759098635s) [noise=true , address=145.79.227.23:4547 , piece_id=MHQLLBRCBITNGVJIWPKGNWFL4EUREXWOFQXWJ63677TYZCUVTHWQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.899900451 (621h38m39.770983486s) [noise=true , address=145.79.227.41:4087 , piece_id=4NAHQPXVCOXNK4C6TB5XMEE2ODSMGZORRRSTOMQM2NO7RQ7P6RUA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.899937080 (621h38m39.771020115s) [noise=true , address=145.79.227.15:4875 , piece_id=6AE5CPDQXRGPVBOHPCTJ6H7Y4T7IS6SH5FJ7E6C7RLMFLQRR24MA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.909266034 (621h38m39.780349069s) [noise=true , address=145.79.227.39:4013 , piece_id=MZ6WK2YSN5YHAT5VZ7D4TXCVJZEHJNBW2IE2IIJKO67H6TZ46IUQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:39.909923637 (621h38m39.781006682s) [noise=true , address=145.79.227.5:4221 , piece_id=QPEXTQN54JUWBHXNBHNL3PXLJ6ZCYQ4MTMW6667UGVMU7HZJBS6Q, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:40.150148398 (621h38m40.021231443s) [noise=true , address=45.91.116.10:4082 , piece_id=6GQV72QGXNBRH53RUWSZIIGO5AA6SJZA6BSDO7AKKG25Z24EXNWQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:40.154041841 (621h38m40.025124876s) [noise=true , address=45.91.116.12:4227 , piece_id=UMU3YBLMKL5TA3I4XZYD6JFFJ4AMHN2FZ3NB227QN3LP3S3KMXHA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:40.168507246 (621h38m40.039590291s) [noise=true , address=45.91.116.9:4261 , piece_id=XOPXTR3GXDMYDO22TBBKES46RZMXUYBRLB6LLZYMJMAUTHPAKHIQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: <nil>
Dec 31 13:07:40.168527746 (621h38m40.039610781s) [seg_pos={0 0}, upload=544735]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:07:40.168737756 (621h38m40.039820791s) [noise=true , address=77.90.31.13:4626 , piece_id=2ELYH6QQQDTIRJ4PV4MPMSA2QHBMIOAFTDMGPN7M5EBTESFWZU6Q, seg_pos={0 0}, upload=544735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 31 13:07:40.168744188 (621h38m40.039827223s) [noise=true , address=77.90.31.13:4626 , piece_id=2ELYH6QQQDTIRJ4PV4MPMSA2QHBMIOAFTDMGPN7M5EBTESFWZU6Q, seg_pos={0 0}, upload=544735]: Long tail cancellation due to optimal threshold
Dec 31 13:07:40.168865698 (621h38m40.039948734s) [noise=true , address=145.79.227.34:4905 , piece_id=DOU3H2BQNW7HKQ5VZ5MVUOHHWF6OD67HOG3ACWUGBT7IM3Q6LCHA, seg_pos={0 0}, upload=544735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 31 13:07:40.168882441 (621h38m40.039965476s) [noise=true , address=145.79.227.34:4905 , piece_id=DOU3H2BQNW7HKQ5VZ5MVUOHHWF6OD67HOG3ACWUGBT7IM3Q6LCHA, seg_pos={0 0}, upload=544735]: Long tail cancellation due to optimal threshold
Dec 31 13:07:40.169044589 (621h38m40.040127634s) [noise=true , address=77.90.31.10:4520 , piece_id=5T4L2XHK42SW2VBVE2YI2XBNOI6SD2QPWTXBU6RYQYFNFFT7AAEQ, seg_pos={0 0}, upload=544735]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 31 13:07:40.169053186 (621h38m40.040136231s) [noise=true , address=77.90.31.10:4520 , piece_id=5T4L2XHK42SW2VBVE2YI2XBNOI6SD2QPWTXBU6RYQYFNFFT7AAEQ, seg_pos={0 0}, upload=544735]: Long tail cancellation due to optimal threshold
Dec 31 13:07:40.169074887 (621h38m40.040157922s) [seg_pos={0 0}, upload=544735]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:07:40.169147224 (621h38m40.04023026s) [upload=544735]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:07:40.169152686 (621h38m40.040235721s) [upload=544735]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:07:40.464018252 (621h38m40.335101287s) [upload=544735]: Done uploading segments...
Dec 31 13:07:40.464039242 (621h38m40.335122277s) [upload=544735]: Upload finished. err: <nil>
Duration: 30.00418385s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
2.919274526s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:07:54 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 2.919300626s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [........X.................X.................................] duration:
39.187878298s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:07:51 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 39.18790551s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................................X.....X..] duration:
26.458408589s trace:
https://jaeger.dev.storj.io/trace/2513d132f2ff5c93
FAIL (2025-12-31 13:07:39 UTC)
orbiter warning: context deadline exceeded
Duration: 26.458433105s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................X...................................] duration:
3.640031781s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:07:37 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.640070365s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [XXX..X...X.......X......X.X...X.X........XX..X.X..X....XX..X] duration:
30.003246328s trace:
https://jaeger.dev.storj.io/trace/15255eccd818ac13
FAIL (2025-12-31 13:07:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:06:44.268969823 (621h37m44.140052858s) [upload=544724]: Starting upload
Dec 31 13:06:44.269070685 (621h37m44.14015372s) [upload=544724]: Done starting upload
Dec 31 13:06:44.269113085 (621h37m44.14019612s) [upload=544724]: Uploading segments...
Dec 31 13:06:44.269240979 (621h37m44.140324014s) [upload=544724]: Got next segment. Inline: false
Dec 31 13:06:44.269271907 (621h37m44.140354942s) [upload=544724]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:06:44.269276216 (621h37m44.140359251s) [upload=544724]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:06:44.522342441 (621h37m44.393425476s) [seg_pos={0 0}, upload=544724]: Begin upload segment...
Dec 31 13:06:44.522399610 (621h37m44.393482655s) [seg_pos={0 0}, upload=544724]: Done begin upload segment.
Dec 31 13:06:44.522438655 (621h37m44.39352169s) [noise=true , address=77.90.31.14:4664 , piece_id=22NUQLADE2OKYYFSTJIYTHHXJ3NT5QEFFS76WV2XQFO65WD4RAQQ, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.522492878 (621h37m44.393575923s) [noise=true , address=145.79.227.11:4729 , piece_id=AIYH4ROAGN5MG6QFBUENCNSONG6C5YL55NHIGXDG366ZEYJUOBPA, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.523031103 (621h37m44.394114148s) [noise=true , address=145.79.227.47:4267 , piece_id=3WN3IBCBTPW3VNFZSOK7KLPXSIIN7VWVYEFSL2QJX5445D3SUDMA, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.523394514 (621h37m44.39447756s) [noise=true , address=145.79.227.5:4223 , piece_id=B5BYXHO5E6V665RGHQGT2YY7URMTYAD4FCELMARAUUKE4PARXAIA, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.523853047 (621h37m44.394936082s) [noise=true , address=145.79.227.7:4293 , piece_id=TXSY4CZYBWNZ2X5JJI4L2II4NTQMPXB74XRNKR4WOCIAPDX2DCYQ, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.524174910 (621h37m44.395257945s) [noise=true , address=145.79.227.3:4150 , piece_id=JRGV4L33XQJRY5XU65V4LKG6DN5SMSHBSFPYRCGRRV5GVDCL3J2Q, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.524516000 (621h37m44.395599035s) [noise=true , address=145.79.227.18:4006 , piece_id=VVXJAU4Z72V5CNKFSN3DCQGQUTBPDGRSLIUICXDNHOR7DE23UMGQ, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.524842242 (621h37m44.395925287s) [noise=true , address=77.90.31.10:4519 , piece_id=TJQEQMSUETSRGP5U2DVUG57V67JLQ6DYA3J26HUXNGICRSQIIFPA, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.525183220 (621h37m44.396266255s) [noise=true , address=145.79.227.19:4077 , piece_id=B36BKJ4O6L2DV6L3GTVD3QPT3NP2SB73L5FLDL54MICIB72KOLIA, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.525583964 (621h37m44.396667009s) [noise=true , address=77.90.31.13:4627 , piece_id=62IWLKIBX635Y4U44XYHT3N77PVYN524TRGMZ7MO5BQGNXKAV4XA, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.526826380 (621h37m44.397909425s) [noise=true , address=45.91.116.9:4261 , piece_id=6ECRADMRXK3WY3HXKLS5BKDRZKYM3EWZQSZ6OTS4WG7CYKOI3KSA, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.526867828 (621h37m44.397950874s) [noise=true , address=145.79.227.17:4970 , piece_id=6DMD2EFDAK7SOEDNPN3HI3BMNFLOIZBWQFHTC7NTFFSGTAHLLXJQ, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.528142034 (621h37m44.399225069s) [noise=true , address=145.79.227.14:4838 , piece_id=T4RZB44IU6R4CYZ2MCY6S2KREZM2HVPEZVSEBZLENNFO763777XQ, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.529374320 (621h37m44.400457355s) [noise=true , address=145.79.227.23:4547 , piece_id=CXPD46HRWDOBADLQDWRNE5P6TZMWXHBIJM7YYG77MQUI2FCCWQQQ, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.530751122 (621h37m44.401834167s) [noise=true , address=77.90.31.11:4556 , piece_id=L35GT5ZUSXGCWVA2GT52TFPM2VYJJHZL4HRHDIQGQO6P2XFR6JTA, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:44.531935608 (621h37m44.403018653s) [noise=true , address=45.91.116.13:4154 , piece_id=JXXDBMRGTUPRW6STUYF3554HMJDPV3XZWHA3NOW55WTTXN5Y7FHA, seg_pos={0 0}, upload=544724]: Uploading piece...
Dec 31 13:06:46.236859358 (621h37m46.107942393s) [upload=544724]: Next returned nil segment
Dec 31 13:06:46.236872624 (621h37m46.107955669s) [upload=544724]: Waiting for error group managing segments...
Dec 31 13:06:46.515714238 (621h37m46.386797283s) [noise=true , address=145.79.227.19:4077 , piece_id=B36BKJ4O6L2DV6L3GTVD3QPT3NP2SB73L5FLDL54MICIB72KOLIA, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.521131443 (621h37m46.392214478s) [noise=true , address=145.79.227.11:4729 , piece_id=AIYH4ROAGN5MG6QFBUENCNSONG6C5YL55NHIGXDG366ZEYJUOBPA, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.522699519 (621h37m46.393782554s) [noise=true , address=145.79.227.18:4006 , piece_id=VVXJAU4Z72V5CNKFSN3DCQGQUTBPDGRSLIUICXDNHOR7DE23UMGQ, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.527149272 (621h37m46.398232307s) [noise=true , address=145.79.227.23:4547 , piece_id=CXPD46HRWDOBADLQDWRNE5P6TZMWXHBIJM7YYG77MQUI2FCCWQQQ, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.530193989 (621h37m46.401277034s) [noise=true , address=145.79.227.3:4150 , piece_id=JRGV4L33XQJRY5XU65V4LKG6DN5SMSHBSFPYRCGRRV5GVDCL3J2Q, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.534704098 (621h37m46.405787133s) [noise=true , address=145.79.227.5:4223 , piece_id=B5BYXHO5E6V665RGHQGT2YY7URMTYAD4FCELMARAUUKE4PARXAIA, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.536490970 (621h37m46.407574005s) [noise=true , address=145.79.227.17:4970 , piece_id=6DMD2EFDAK7SOEDNPN3HI3BMNFLOIZBWQFHTC7NTFFSGTAHLLXJQ, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.548794618 (621h37m46.419877663s) [noise=true , address=145.79.227.14:4838 , piece_id=T4RZB44IU6R4CYZ2MCY6S2KREZM2HVPEZVSEBZLENNFO763777XQ, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.564495355 (621h37m46.43557839s) [noise=true , address=145.79.227.47:4267 , piece_id=3WN3IBCBTPW3VNFZSOK7KLPXSIIN7VWVYEFSL2QJX5445D3SUDMA, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.604637511 (621h37m46.475720546s) [noise=true , address=145.79.227.7:4293 , piece_id=TXSY4CZYBWNZ2X5JJI4L2II4NTQMPXB74XRNKR4WOCIAPDX2DCYQ, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.713369232 (621h37m46.584452267s) [noise=true , address=77.90.31.10:4519 , piece_id=TJQEQMSUETSRGP5U2DVUG57V67JLQ6DYA3J26HUXNGICRSQIIFPA, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.798560618 (621h37m46.669643664s) [noise=true , address=77.90.31.14:4664 , piece_id=22NUQLADE2OKYYFSTJIYTHHXJ3NT5QEFFS76WV2XQFO65WD4RAQQ, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.806290578 (621h37m46.677373613s) [noise=true , address=77.90.31.11:4556 , piece_id=L35GT5ZUSXGCWVA2GT52TFPM2VYJJHZL4HRHDIQGQO6P2XFR6JTA, seg_pos={0 0}, upload=544724]: Done uploading piece. err: <nil>
Dec 31 13:06:46.806310616 (621h37m46.677393651s) [seg_pos={0 0}, upload=544724]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:06:46.806596190 (621h37m46.677679225s) [noise=true , address=45.91.116.9:4261 , piece_id=6ECRADMRXK3WY3HXKLS5BKDRZKYM3EWZQSZ6OTS4WG7CYKOI3KSA, seg_pos={0 0}, upload=544724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 31 13:06:46.806618822 (621h37m46.677701887s) [noise=true , address=45.91.116.9:4261 , piece_id=6ECRADMRXK3WY3HXKLS5BKDRZKYM3EWZQSZ6OTS4WG7CYKOI3KSA, seg_pos={0 0}, upload=544724]: Long tail cancellation due to optimal threshold
Dec 31 13:06:46.806653889 (621h37m46.677736924s) [noise=true , address=77.90.31.13:4627 , piece_id=62IWLKIBX635Y4U44XYHT3N77PVYN524TRGMZ7MO5BQGNXKAV4XA, seg_pos={0 0}, upload=544724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:53172->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 31 13:06:46.806662045 (621h37m46.67774508s) [noise=true , address=77.90.31.13:4627 , piece_id=62IWLKIBX635Y4U44XYHT3N77PVYN524TRGMZ7MO5BQGNXKAV4XA, seg_pos={0 0}, upload=544724]: Long tail cancellation due to optimal threshold
Dec 31 13:06:46.806960794 (621h37m46.678043859s) [noise=true , address=45.91.116.13:4154 , piece_id=JXXDBMRGTUPRW6STUYF3554HMJDPV3XZWHA3NOW55WTTXN5Y7FHA, seg_pos={0 0}, upload=544724]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 31 13:06:46.806970351 (621h37m46.678053387s) [noise=true , address=45.91.116.13:4154 , piece_id=JXXDBMRGTUPRW6STUYF3554HMJDPV3XZWHA3NOW55WTTXN5Y7FHA, seg_pos={0 0}, upload=544724]: Long tail cancellation due to optimal threshold
Dec 31 13:06:46.806992444 (621h37m46.678075479s) [seg_pos={0 0}, upload=544724]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:06:46.807076434 (621h37m46.67815948s) [upload=544724]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:06:46.807083678 (621h37m46.678166713s) [upload=544724]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:06:47.103745644 (621h37m46.97482868s) [upload=544724]: Done uploading segments...
Dec 31 13:06:47.103762086 (621h37m46.974845121s) [upload=544724]: Upload finished. err: <nil>
Duration: 30.003271666s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [X........................X..................................] duration:
3.782723601s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:07:06 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.782748648s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........X.................X................................] duration:
29.84692606s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:07:05 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 29.846950899s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X.....X.] duration:
13.539131933s trace:
https://jaeger.dev.storj.io/trace/2513d132f2ff5c93
FAIL (2025-12-31 13:07:02 UTC)
orbiter warning: context deadline exceeded
Duration: 13.539316244s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
5.00727783s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:06:49 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.007304982s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.X........................X.................................] duration:
4.140287332s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:06:42 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.14031305s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [XXXX..X...X.......X......X.X...X.X........XX..X.X..X....XX..] duration:
24.883825396s trace:
https://jaeger.dev.storj.io/trace/4d7c74413b69f679
FAIL (2025-12-31 13:06:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 7622VBLGLFS7JXUSIBOBD3J4TVWHRTVZEXUI4OBPTAE6WAMYGBTA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: 66GG2JAYBPKO4Z52UASX2NGTLMWQHWBQG53ZP5HZJB6CMKJUBH5Q) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: RZXLQGSCMWIIXJ45UAPNX3D4UP2U7ATZJLBBIPFDYYZSCR2YF2XQ) context deadline exceeded
======LOGS======
Dec 31 13:03:53.287358847 (621h34m53.158441882s) [upload=544685]: Starting upload
Dec 31 13:03:53.287432306 (621h34m53.158515342s) [upload=544685]: Done starting upload
Dec 31 13:03:53.287623941 (621h34m53.158706986s) [upload=544685]: Uploading segments...
Dec 31 13:03:53.287944491 (621h34m53.159027536s) [upload=544685]: Got next segment. Inline: false
Dec 31 13:03:53.288021008 (621h34m53.159104043s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:03:53.288027951 (621h34m53.159110996s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:03:53.532441419 (621h34m53.403524464s) [seg_pos={0 0}, upload=544685]: Begin upload segment...
Dec 31 13:03:53.532501774 (621h34m53.403584819s) [seg_pos={0 0}, upload=544685]: Done begin upload segment.
Dec 31 13:03:53.532554945 (621h34m53.40363799s) [noise=true , address=145.79.227.4:4187 , piece_id=X3CHLA2GO42FLVDPYDVCRNXBC3QKLQUO7RGQSA5CJKXDAPP3BDEA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.532571637 (621h34m53.403654682s) [noise=true , address=145.79.227.20:4437 , piece_id=LTTCG76BSEGDLUAEY2HOETH2VLDCDDXZXD7OPJ2WX3Y53JZLKJCQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.533044577 (621h34m53.404127612s) [noise=true , address=77.90.31.13:4629 , piece_id=F2JRQ4COYUDPGR2EYHG77R4XHOOW4VZPHFYRDPE4P2XZRY4PIWFQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.533477452 (621h34m53.404560497s) [noise=true , address=145.79.227.2:4113 , piece_id=GEL3MYGMSAWTAJOKAJWJAI7MQAA7P4RF24G5ZFOTEW7UCZSNSFXQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.534000728 (621h34m53.405083773s) [noise=true , address=145.79.227.47:4267 , piece_id=3TLC7V6E7MPNJTC6Q3A7P5SDL6ETUCETX365YPTUMRBM3LTBN47A, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.534443701 (621h34m53.405526746s) [noise=true , address=145.79.227.51:4409 , piece_id=RFV3JSSCGNMKT6IACDS6TJ6KDPGL3CRQ4PFW3POOEFA4645GGKWQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.534926050 (621h34m53.406009095s) [noise=true , address=145.79.227.43:4159 , piece_id=GOWJ6637NR64ZVWLEUBP7ZWDTSHP555HFOPSH4K5VLGUFAMRZTNA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.535452482 (621h34m53.406535527s) [noise=true , address=77.90.31.11:4554 , piece_id=XZXBGTUJAXPBQDILL2XNZREEMEQXKAJXTI5WTKPTXQXTRJ7B7K7Q, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.536954532 (621h34m53.408037567s) [noise=true , address=77.90.31.16:4735 , piece_id=7622VBLGLFS7JXUSIBOBD3J4TVWHRTVZEXUI4OBPTAE6WAMYGBTA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.537258421 (621h34m53.408341456s) [noise=true , address=77.90.31.17:4773 , piece_id=Y247CJQP3IRTUMSFT3AI5HOBPXB3QIILDCQLJN3C5Z5KLAEAF7LA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.538249659 (621h34m53.409332694s) [noise=true , address=145.79.227.33:4871 , piece_id=4VILV5AFTA7JQALXVME6QG4NZPGHVLEKLWXCHXQ7UTCR2VQFUFNA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.539517783 (621h34m53.410600818s) [noise=true , address=45.91.116.9:4264 , piece_id=XW3IMKSUXOVOV53H4TEEWASY2Z2WFO22PNVDD35YOAA4UBIQCJYQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.540872302 (621h34m53.411955338s) [noise=true , address=145.79.227.12:4767 , piece_id=SRQFYAIN3RBZJX3OHLYNP5CVKBMZLJHNI5OZQ3XXKOULPV6RIB6Q, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.542299360 (621h34m53.413382405s) [noise=true , address=77.90.31.21:4878 , piece_id=66GG2JAYBPKO4Z52UASX2NGTLMWQHWBQG53ZP5HZJB6CMKJUBH5Q, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.544031068 (621h34m53.415114103s) [noise=true , address=145.79.227.26:4654 , piece_id=OWSOFG2IOVX2G76T62CC27XBKQ5FYIZPKQLHBVECCZL4SJRF56PA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.546199286 (621h34m53.417282331s) [noise=true , address=45.91.116.12:4226 , piece_id=RZXLQGSCMWIIXJ45UAPNX3D4UP2U7ATZJLBBIPFDYYZSCR2YF2XQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:55.244986602 (621h34m55.116069647s) [upload=544685]: Next returned nil segment
Dec 31 13:03:55.245000879 (621h34m55.116083915s) [upload=544685]: Waiting for error group managing segments...
Dec 31 13:03:55.519546022 (621h34m55.390629067s) [noise=true , address=145.79.227.26:4654 , piece_id=OWSOFG2IOVX2G76T62CC27XBKQ5FYIZPKQLHBVECCZL4SJRF56PA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.534091840 (621h34m55.405174875s) [noise=true , address=145.79.227.20:4437 , piece_id=LTTCG76BSEGDLUAEY2HOETH2VLDCDDXZXD7OPJ2WX3Y53JZLKJCQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.539004404 (621h34m55.410087439s) [noise=true , address=145.79.227.47:4267 , piece_id=3TLC7V6E7MPNJTC6Q3A7P5SDL6ETUCETX365YPTUMRBM3LTBN47A, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.542986126 (621h34m55.414069161s) [noise=true , address=145.79.227.51:4409 , piece_id=RFV3JSSCGNMKT6IACDS6TJ6KDPGL3CRQ4PFW3POOEFA4645GGKWQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.545267169 (621h34m55.416350214s) [noise=true , address=145.79.227.2:4113 , piece_id=GEL3MYGMSAWTAJOKAJWJAI7MQAA7P4RF24G5ZFOTEW7UCZSNSFXQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.548506327 (621h34m55.419589362s) [noise=true , address=145.79.227.12:4767 , piece_id=SRQFYAIN3RBZJX3OHLYNP5CVKBMZLJHNI5OZQ3XXKOULPV6RIB6Q, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.609082122 (621h34m55.480165157s) [noise=true , address=145.79.227.4:4187 , piece_id=X3CHLA2GO42FLVDPYDVCRNXBC3QKLQUO7RGQSA5CJKXDAPP3BDEA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.699328925 (621h34m55.57041196s) [noise=true , address=77.90.31.13:4629 , piece_id=F2JRQ4COYUDPGR2EYHG77R4XHOOW4VZPHFYRDPE4P2XZRY4PIWFQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.717850953 (621h34m55.588933988s) [noise=true , address=77.90.31.16:4735 , piece_id=7622VBLGLFS7JXUSIBOBD3J4TVWHRTVZEXUI4OBPTAE6WAMYGBTA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.736819563 (621h34m55.607902598s) [noise=true , address=145.79.227.43:4159 , piece_id=GOWJ6637NR64ZVWLEUBP7ZWDTSHP555HFOPSH4K5VLGUFAMRZTNA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.764173479 (621h34m55.635256514s) [noise=true , address=77.90.31.21:4878 , piece_id=66GG2JAYBPKO4Z52UASX2NGTLMWQHWBQG53ZP5HZJB6CMKJUBH5Q, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.861244005 (621h34m55.73232704s) [noise=true , address=45.91.116.12:4226 , piece_id=RZXLQGSCMWIIXJ45UAPNX3D4UP2U7ATZJLBBIPFDYYZSCR2YF2XQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:56.451322650 (621h34m56.322405695s) [noise=true , address=77.90.31.11:4554 , piece_id=XZXBGTUJAXPBQDILL2XNZREEMEQXKAJXTI5WTKPTXQXTRJ7B7K7Q, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:56.451348098 (621h34m56.322431143s) [seg_pos={0 0}, upload=544685]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:03:56.451662387 (621h34m56.322745422s) [noise=true , address=77.90.31.17:4773 , piece_id=Y247CJQP3IRTUMSFT3AI5HOBPXB3QIILDCQLJN3C5Z5KLAEAF7LA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:56.451670121 (621h34m56.322753176s) [noise=true , address=77.90.31.17:4773 , piece_id=Y247CJQP3IRTUMSFT3AI5HOBPXB3QIILDCQLJN3C5Z5KLAEAF7LA, seg_pos={0 0}, upload=544685]: Long tail cancellation due to optimal threshold
Dec 31 13:03:56.451823003 (621h34m56.322906038s) [noise=true , address=145.79.227.33:4871 , piece_id=4VILV5AFTA7JQALXVME6QG4NZPGHVLEKLWXCHXQ7UTCR2VQFUFNA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:56.451829856 (621h34m56.322912901s) [noise=true , address=145.79.227.33:4871 , piece_id=4VILV5AFTA7JQALXVME6QG4NZPGHVLEKLWXCHXQ7UTCR2VQFUFNA, seg_pos={0 0}, upload=544685]: Long tail cancellation due to optimal threshold
Dec 31 13:03:56.451861226 (621h34m56.322944271s) [noise=true , address=45.91.116.9:4264 , piece_id=XW3IMKSUXOVOV53H4TEEWASY2Z2WFO22PNVDD35YOAA4UBIQCJYQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:56.451899579 (621h34m56.322982614s) [noise=true , address=45.91.116.9:4264 , piece_id=XW3IMKSUXOVOV53H4TEEWASY2Z2WFO22PNVDD35YOAA4UBIQCJYQ, seg_pos={0 0}, upload=544685]: Long tail cancellation due to optimal threshold
Dec 31 13:03:56.451923414 (621h34m56.323006459s) [seg_pos={0 0}, upload=544685]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:03:56.452013205 (621h34m56.32309624s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:03:56.452019266 (621h34m56.323102301s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:03:56.752058034 (621h34m56.623141069s) [upload=544685]: Done uploading segments...
Dec 31 13:03:56.752081758 (621h34m56.623164793s) [upload=544685]: Upload finished. err: <nil>
Duration: 24.883852287s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X.....X] duration:
30.000907091s trace:
https://jaeger.dev.storj.io/trace/2513d132f2ff5c93
FAIL (2025-12-31 13:06:31 UTC)
orbiter warning: context deadline exceeded
Duration: 30.00100592s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X.................X...............................] duration:
36.611968733s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:06:20 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 36.611987068s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
3.88591973s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:05:59 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 3.885946721s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........X.................X..............................] duration:
38.848810764s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:05:43 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 38.848842615s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [X.X........................X................................] duration:
3.78966713s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:05:37 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.789694914s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
7.704011649s trace:
https://jaeger.dev.storj.io/trace/51ca50ba1c5675ff
FAIL (2025-12-31 13:05:17 UTC)
orbiter warning: context deadline exceeded
Duration: 7.704039831s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.XXXX..X...X.......X......X.X...X.X........XX..X.X..X....XX.] duration:
10.793201475s trace:
https://jaeger.dev.storj.io/trace/4d7c74413b69f679
FAIL (2025-12-31 13:05:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 7622VBLGLFS7JXUSIBOBD3J4TVWHRTVZEXUI4OBPTAE6WAMYGBTA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: 66GG2JAYBPKO4Z52UASX2NGTLMWQHWBQG53ZP5HZJB6CMKJUBH5Q) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: RZXLQGSCMWIIXJ45UAPNX3D4UP2U7ATZJLBBIPFDYYZSCR2YF2XQ) context deadline exceeded
======LOGS======
Dec 31 13:03:53.287358847 (621h34m53.158441882s) [upload=544685]: Starting upload
Dec 31 13:03:53.287432306 (621h34m53.158515342s) [upload=544685]: Done starting upload
Dec 31 13:03:53.287623941 (621h34m53.158706986s) [upload=544685]: Uploading segments...
Dec 31 13:03:53.287944491 (621h34m53.159027536s) [upload=544685]: Got next segment. Inline: false
Dec 31 13:03:53.288021008 (621h34m53.159104043s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:03:53.288027951 (621h34m53.159110996s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:03:53.532441419 (621h34m53.403524464s) [seg_pos={0 0}, upload=544685]: Begin upload segment...
Dec 31 13:03:53.532501774 (621h34m53.403584819s) [seg_pos={0 0}, upload=544685]: Done begin upload segment.
Dec 31 13:03:53.532554945 (621h34m53.40363799s) [noise=true , address=145.79.227.4:4187 , piece_id=X3CHLA2GO42FLVDPYDVCRNXBC3QKLQUO7RGQSA5CJKXDAPP3BDEA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.532571637 (621h34m53.403654682s) [noise=true , address=145.79.227.20:4437 , piece_id=LTTCG76BSEGDLUAEY2HOETH2VLDCDDXZXD7OPJ2WX3Y53JZLKJCQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.533044577 (621h34m53.404127612s) [noise=true , address=77.90.31.13:4629 , piece_id=F2JRQ4COYUDPGR2EYHG77R4XHOOW4VZPHFYRDPE4P2XZRY4PIWFQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.533477452 (621h34m53.404560497s) [noise=true , address=145.79.227.2:4113 , piece_id=GEL3MYGMSAWTAJOKAJWJAI7MQAA7P4RF24G5ZFOTEW7UCZSNSFXQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.534000728 (621h34m53.405083773s) [noise=true , address=145.79.227.47:4267 , piece_id=3TLC7V6E7MPNJTC6Q3A7P5SDL6ETUCETX365YPTUMRBM3LTBN47A, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.534443701 (621h34m53.405526746s) [noise=true , address=145.79.227.51:4409 , piece_id=RFV3JSSCGNMKT6IACDS6TJ6KDPGL3CRQ4PFW3POOEFA4645GGKWQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.534926050 (621h34m53.406009095s) [noise=true , address=145.79.227.43:4159 , piece_id=GOWJ6637NR64ZVWLEUBP7ZWDTSHP555HFOPSH4K5VLGUFAMRZTNA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.535452482 (621h34m53.406535527s) [noise=true , address=77.90.31.11:4554 , piece_id=XZXBGTUJAXPBQDILL2XNZREEMEQXKAJXTI5WTKPTXQXTRJ7B7K7Q, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.536954532 (621h34m53.408037567s) [noise=true , address=77.90.31.16:4735 , piece_id=7622VBLGLFS7JXUSIBOBD3J4TVWHRTVZEXUI4OBPTAE6WAMYGBTA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.537258421 (621h34m53.408341456s) [noise=true , address=77.90.31.17:4773 , piece_id=Y247CJQP3IRTUMSFT3AI5HOBPXB3QIILDCQLJN3C5Z5KLAEAF7LA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.538249659 (621h34m53.409332694s) [noise=true , address=145.79.227.33:4871 , piece_id=4VILV5AFTA7JQALXVME6QG4NZPGHVLEKLWXCHXQ7UTCR2VQFUFNA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.539517783 (621h34m53.410600818s) [noise=true , address=45.91.116.9:4264 , piece_id=XW3IMKSUXOVOV53H4TEEWASY2Z2WFO22PNVDD35YOAA4UBIQCJYQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.540872302 (621h34m53.411955338s) [noise=true , address=145.79.227.12:4767 , piece_id=SRQFYAIN3RBZJX3OHLYNP5CVKBMZLJHNI5OZQ3XXKOULPV6RIB6Q, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.542299360 (621h34m53.413382405s) [noise=true , address=77.90.31.21:4878 , piece_id=66GG2JAYBPKO4Z52UASX2NGTLMWQHWBQG53ZP5HZJB6CMKJUBH5Q, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.544031068 (621h34m53.415114103s) [noise=true , address=145.79.227.26:4654 , piece_id=OWSOFG2IOVX2G76T62CC27XBKQ5FYIZPKQLHBVECCZL4SJRF56PA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.546199286 (621h34m53.417282331s) [noise=true , address=45.91.116.12:4226 , piece_id=RZXLQGSCMWIIXJ45UAPNX3D4UP2U7ATZJLBBIPFDYYZSCR2YF2XQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:55.244986602 (621h34m55.116069647s) [upload=544685]: Next returned nil segment
Dec 31 13:03:55.245000879 (621h34m55.116083915s) [upload=544685]: Waiting for error group managing segments...
Dec 31 13:03:55.519546022 (621h34m55.390629067s) [noise=true , address=145.79.227.26:4654 , piece_id=OWSOFG2IOVX2G76T62CC27XBKQ5FYIZPKQLHBVECCZL4SJRF56PA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.534091840 (621h34m55.405174875s) [noise=true , address=145.79.227.20:4437 , piece_id=LTTCG76BSEGDLUAEY2HOETH2VLDCDDXZXD7OPJ2WX3Y53JZLKJCQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.539004404 (621h34m55.410087439s) [noise=true , address=145.79.227.47:4267 , piece_id=3TLC7V6E7MPNJTC6Q3A7P5SDL6ETUCETX365YPTUMRBM3LTBN47A, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.542986126 (621h34m55.414069161s) [noise=true , address=145.79.227.51:4409 , piece_id=RFV3JSSCGNMKT6IACDS6TJ6KDPGL3CRQ4PFW3POOEFA4645GGKWQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.545267169 (621h34m55.416350214s) [noise=true , address=145.79.227.2:4113 , piece_id=GEL3MYGMSAWTAJOKAJWJAI7MQAA7P4RF24G5ZFOTEW7UCZSNSFXQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.548506327 (621h34m55.419589362s) [noise=true , address=145.79.227.12:4767 , piece_id=SRQFYAIN3RBZJX3OHLYNP5CVKBMZLJHNI5OZQ3XXKOULPV6RIB6Q, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.609082122 (621h34m55.480165157s) [noise=true , address=145.79.227.4:4187 , piece_id=X3CHLA2GO42FLVDPYDVCRNXBC3QKLQUO7RGQSA5CJKXDAPP3BDEA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.699328925 (621h34m55.57041196s) [noise=true , address=77.90.31.13:4629 , piece_id=F2JRQ4COYUDPGR2EYHG77R4XHOOW4VZPHFYRDPE4P2XZRY4PIWFQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.717850953 (621h34m55.588933988s) [noise=true , address=77.90.31.16:4735 , piece_id=7622VBLGLFS7JXUSIBOBD3J4TVWHRTVZEXUI4OBPTAE6WAMYGBTA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.736819563 (621h34m55.607902598s) [noise=true , address=145.79.227.43:4159 , piece_id=GOWJ6637NR64ZVWLEUBP7ZWDTSHP555HFOPSH4K5VLGUFAMRZTNA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.764173479 (621h34m55.635256514s) [noise=true , address=77.90.31.21:4878 , piece_id=66GG2JAYBPKO4Z52UASX2NGTLMWQHWBQG53ZP5HZJB6CMKJUBH5Q, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.861244005 (621h34m55.73232704s) [noise=true , address=45.91.116.12:4226 , piece_id=RZXLQGSCMWIIXJ45UAPNX3D4UP2U7ATZJLBBIPFDYYZSCR2YF2XQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:56.451322650 (621h34m56.322405695s) [noise=true , address=77.90.31.11:4554 , piece_id=XZXBGTUJAXPBQDILL2XNZREEMEQXKAJXTI5WTKPTXQXTRJ7B7K7Q, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:56.451348098 (621h34m56.322431143s) [seg_pos={0 0}, upload=544685]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:03:56.451662387 (621h34m56.322745422s) [noise=true , address=77.90.31.17:4773 , piece_id=Y247CJQP3IRTUMSFT3AI5HOBPXB3QIILDCQLJN3C5Z5KLAEAF7LA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:56.451670121 (621h34m56.322753176s) [noise=true , address=77.90.31.17:4773 , piece_id=Y247CJQP3IRTUMSFT3AI5HOBPXB3QIILDCQLJN3C5Z5KLAEAF7LA, seg_pos={0 0}, upload=544685]: Long tail cancellation due to optimal threshold
Dec 31 13:03:56.451823003 (621h34m56.322906038s) [noise=true , address=145.79.227.33:4871 , piece_id=4VILV5AFTA7JQALXVME6QG4NZPGHVLEKLWXCHXQ7UTCR2VQFUFNA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:56.451829856 (621h34m56.322912901s) [noise=true , address=145.79.227.33:4871 , piece_id=4VILV5AFTA7JQALXVME6QG4NZPGHVLEKLWXCHXQ7UTCR2VQFUFNA, seg_pos={0 0}, upload=544685]: Long tail cancellation due to optimal threshold
Dec 31 13:03:56.451861226 (621h34m56.322944271s) [noise=true , address=45.91.116.9:4264 , piece_id=XW3IMKSUXOVOV53H4TEEWASY2Z2WFO22PNVDD35YOAA4UBIQCJYQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:56.451899579 (621h34m56.322982614s) [noise=true , address=45.91.116.9:4264 , piece_id=XW3IMKSUXOVOV53H4TEEWASY2Z2WFO22PNVDD35YOAA4UBIQCJYQ, seg_pos={0 0}, upload=544685]: Long tail cancellation due to optimal threshold
Dec 31 13:03:56.451923414 (621h34m56.323006459s) [seg_pos={0 0}, upload=544685]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:03:56.452013205 (621h34m56.32309624s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:03:56.452019266 (621h34m56.323102301s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:03:56.752058034 (621h34m56.623141069s) [upload=544685]: Done uploading segments...
Dec 31 13:03:56.752081758 (621h34m56.623164793s) [upload=544685]: Upload finished. err: <nil>
Duration: 10.79323605s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.X.X........................X...............................] duration:
3.680136668s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:04:55 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.680164201s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
7.528460071s trace:
https://jaeger.dev.storj.io/trace/51ca50ba1c5675ff
FAIL (2025-12-31 13:04:51 UTC)
orbiter warning: context deadline exceeded
Duration: 7.528552498s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
4.699946646s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:04:48 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 4.699975391s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [............X.................X.............................] duration:
15.395388607s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:04:43 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 15.395406801s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [..X.X........................X..............................] duration:
3.71091074s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:04:36 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.71092654s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............X.................X............................] duration:
36.282382967s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:04:27 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 36.282402805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.XXXX..X...X.......X......X.X...X.X........XX..X.X..X....XX] duration:
30.004705174s trace:
https://jaeger.dev.storj.io/trace/4d7c74413b69f679
FAIL (2025-12-31 13:04:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: 7622VBLGLFS7JXUSIBOBD3J4TVWHRTVZEXUI4OBPTAE6WAMYGBTA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: 66GG2JAYBPKO4Z52UASX2NGTLMWQHWBQG53ZP5HZJB6CMKJUBH5Q) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: RZXLQGSCMWIIXJ45UAPNX3D4UP2U7ATZJLBBIPFDYYZSCR2YF2XQ) context deadline exceeded
======LOGS======
Dec 31 13:03:53.287358847 (621h34m53.158441882s) [upload=544685]: Starting upload
Dec 31 13:03:53.287432306 (621h34m53.158515342s) [upload=544685]: Done starting upload
Dec 31 13:03:53.287623941 (621h34m53.158706986s) [upload=544685]: Uploading segments...
Dec 31 13:03:53.287944491 (621h34m53.159027536s) [upload=544685]: Got next segment. Inline: false
Dec 31 13:03:53.288021008 (621h34m53.159104043s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:03:53.288027951 (621h34m53.159110996s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:03:53.532441419 (621h34m53.403524464s) [seg_pos={0 0}, upload=544685]: Begin upload segment...
Dec 31 13:03:53.532501774 (621h34m53.403584819s) [seg_pos={0 0}, upload=544685]: Done begin upload segment.
Dec 31 13:03:53.532554945 (621h34m53.40363799s) [noise=true , address=145.79.227.4:4187 , piece_id=X3CHLA2GO42FLVDPYDVCRNXBC3QKLQUO7RGQSA5CJKXDAPP3BDEA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.532571637 (621h34m53.403654682s) [noise=true , address=145.79.227.20:4437 , piece_id=LTTCG76BSEGDLUAEY2HOETH2VLDCDDXZXD7OPJ2WX3Y53JZLKJCQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.533044577 (621h34m53.404127612s) [noise=true , address=77.90.31.13:4629 , piece_id=F2JRQ4COYUDPGR2EYHG77R4XHOOW4VZPHFYRDPE4P2XZRY4PIWFQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.533477452 (621h34m53.404560497s) [noise=true , address=145.79.227.2:4113 , piece_id=GEL3MYGMSAWTAJOKAJWJAI7MQAA7P4RF24G5ZFOTEW7UCZSNSFXQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.534000728 (621h34m53.405083773s) [noise=true , address=145.79.227.47:4267 , piece_id=3TLC7V6E7MPNJTC6Q3A7P5SDL6ETUCETX365YPTUMRBM3LTBN47A, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.534443701 (621h34m53.405526746s) [noise=true , address=145.79.227.51:4409 , piece_id=RFV3JSSCGNMKT6IACDS6TJ6KDPGL3CRQ4PFW3POOEFA4645GGKWQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.534926050 (621h34m53.406009095s) [noise=true , address=145.79.227.43:4159 , piece_id=GOWJ6637NR64ZVWLEUBP7ZWDTSHP555HFOPSH4K5VLGUFAMRZTNA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.535452482 (621h34m53.406535527s) [noise=true , address=77.90.31.11:4554 , piece_id=XZXBGTUJAXPBQDILL2XNZREEMEQXKAJXTI5WTKPTXQXTRJ7B7K7Q, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.536954532 (621h34m53.408037567s) [noise=true , address=77.90.31.16:4735 , piece_id=7622VBLGLFS7JXUSIBOBD3J4TVWHRTVZEXUI4OBPTAE6WAMYGBTA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.537258421 (621h34m53.408341456s) [noise=true , address=77.90.31.17:4773 , piece_id=Y247CJQP3IRTUMSFT3AI5HOBPXB3QIILDCQLJN3C5Z5KLAEAF7LA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.538249659 (621h34m53.409332694s) [noise=true , address=145.79.227.33:4871 , piece_id=4VILV5AFTA7JQALXVME6QG4NZPGHVLEKLWXCHXQ7UTCR2VQFUFNA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.539517783 (621h34m53.410600818s) [noise=true , address=45.91.116.9:4264 , piece_id=XW3IMKSUXOVOV53H4TEEWASY2Z2WFO22PNVDD35YOAA4UBIQCJYQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.540872302 (621h34m53.411955338s) [noise=true , address=145.79.227.12:4767 , piece_id=SRQFYAIN3RBZJX3OHLYNP5CVKBMZLJHNI5OZQ3XXKOULPV6RIB6Q, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.542299360 (621h34m53.413382405s) [noise=true , address=77.90.31.21:4878 , piece_id=66GG2JAYBPKO4Z52UASX2NGTLMWQHWBQG53ZP5HZJB6CMKJUBH5Q, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.544031068 (621h34m53.415114103s) [noise=true , address=145.79.227.26:4654 , piece_id=OWSOFG2IOVX2G76T62CC27XBKQ5FYIZPKQLHBVECCZL4SJRF56PA, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:53.546199286 (621h34m53.417282331s) [noise=true , address=45.91.116.12:4226 , piece_id=RZXLQGSCMWIIXJ45UAPNX3D4UP2U7ATZJLBBIPFDYYZSCR2YF2XQ, seg_pos={0 0}, upload=544685]: Uploading piece...
Dec 31 13:03:55.244986602 (621h34m55.116069647s) [upload=544685]: Next returned nil segment
Dec 31 13:03:55.245000879 (621h34m55.116083915s) [upload=544685]: Waiting for error group managing segments...
Dec 31 13:03:55.519546022 (621h34m55.390629067s) [noise=true , address=145.79.227.26:4654 , piece_id=OWSOFG2IOVX2G76T62CC27XBKQ5FYIZPKQLHBVECCZL4SJRF56PA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.534091840 (621h34m55.405174875s) [noise=true , address=145.79.227.20:4437 , piece_id=LTTCG76BSEGDLUAEY2HOETH2VLDCDDXZXD7OPJ2WX3Y53JZLKJCQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.539004404 (621h34m55.410087439s) [noise=true , address=145.79.227.47:4267 , piece_id=3TLC7V6E7MPNJTC6Q3A7P5SDL6ETUCETX365YPTUMRBM3LTBN47A, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.542986126 (621h34m55.414069161s) [noise=true , address=145.79.227.51:4409 , piece_id=RFV3JSSCGNMKT6IACDS6TJ6KDPGL3CRQ4PFW3POOEFA4645GGKWQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.545267169 (621h34m55.416350214s) [noise=true , address=145.79.227.2:4113 , piece_id=GEL3MYGMSAWTAJOKAJWJAI7MQAA7P4RF24G5ZFOTEW7UCZSNSFXQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.548506327 (621h34m55.419589362s) [noise=true , address=145.79.227.12:4767 , piece_id=SRQFYAIN3RBZJX3OHLYNP5CVKBMZLJHNI5OZQ3XXKOULPV6RIB6Q, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.609082122 (621h34m55.480165157s) [noise=true , address=145.79.227.4:4187 , piece_id=X3CHLA2GO42FLVDPYDVCRNXBC3QKLQUO7RGQSA5CJKXDAPP3BDEA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.699328925 (621h34m55.57041196s) [noise=true , address=77.90.31.13:4629 , piece_id=F2JRQ4COYUDPGR2EYHG77R4XHOOW4VZPHFYRDPE4P2XZRY4PIWFQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.717850953 (621h34m55.588933988s) [noise=true , address=77.90.31.16:4735 , piece_id=7622VBLGLFS7JXUSIBOBD3J4TVWHRTVZEXUI4OBPTAE6WAMYGBTA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.736819563 (621h34m55.607902598s) [noise=true , address=145.79.227.43:4159 , piece_id=GOWJ6637NR64ZVWLEUBP7ZWDTSHP555HFOPSH4K5VLGUFAMRZTNA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.764173479 (621h34m55.635256514s) [noise=true , address=77.90.31.21:4878 , piece_id=66GG2JAYBPKO4Z52UASX2NGTLMWQHWBQG53ZP5HZJB6CMKJUBH5Q, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:55.861244005 (621h34m55.73232704s) [noise=true , address=45.91.116.12:4226 , piece_id=RZXLQGSCMWIIXJ45UAPNX3D4UP2U7ATZJLBBIPFDYYZSCR2YF2XQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:56.451322650 (621h34m56.322405695s) [noise=true , address=77.90.31.11:4554 , piece_id=XZXBGTUJAXPBQDILL2XNZREEMEQXKAJXTI5WTKPTXQXTRJ7B7K7Q, seg_pos={0 0}, upload=544685]: Done uploading piece. err: <nil>
Dec 31 13:03:56.451348098 (621h34m56.322431143s) [seg_pos={0 0}, upload=544685]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:03:56.451662387 (621h34m56.322745422s) [noise=true , address=77.90.31.17:4773 , piece_id=Y247CJQP3IRTUMSFT3AI5HOBPXB3QIILDCQLJN3C5Z5KLAEAF7LA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:56.451670121 (621h34m56.322753176s) [noise=true , address=77.90.31.17:4773 , piece_id=Y247CJQP3IRTUMSFT3AI5HOBPXB3QIILDCQLJN3C5Z5KLAEAF7LA, seg_pos={0 0}, upload=544685]: Long tail cancellation due to optimal threshold
Dec 31 13:03:56.451823003 (621h34m56.322906038s) [noise=true , address=145.79.227.33:4871 , piece_id=4VILV5AFTA7JQALXVME6QG4NZPGHVLEKLWXCHXQ7UTCR2VQFUFNA, seg_pos={0 0}, upload=544685]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:56.451829856 (621h34m56.322912901s) [noise=true , address=145.79.227.33:4871 , piece_id=4VILV5AFTA7JQALXVME6QG4NZPGHVLEKLWXCHXQ7UTCR2VQFUFNA, seg_pos={0 0}, upload=544685]: Long tail cancellation due to optimal threshold
Dec 31 13:03:56.451861226 (621h34m56.322944271s) [noise=true , address=45.91.116.9:4264 , piece_id=XW3IMKSUXOVOV53H4TEEWASY2Z2WFO22PNVDD35YOAA4UBIQCJYQ, seg_pos={0 0}, upload=544685]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:56.451899579 (621h34m56.322982614s) [noise=true , address=45.91.116.9:4264 , piece_id=XW3IMKSUXOVOV53H4TEEWASY2Z2WFO22PNVDD35YOAA4UBIQCJYQ, seg_pos={0 0}, upload=544685]: Long tail cancellation due to optimal threshold
Dec 31 13:03:56.451923414 (621h34m56.323006459s) [seg_pos={0 0}, upload=544685]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:03:56.452013205 (621h34m56.32309624s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:03:56.452019266 (621h34m56.323102301s) [upload=544685]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:03:56.752058034 (621h34m56.623141069s) [upload=544685]: Done uploading segments...
Dec 31 13:03:56.752081758 (621h34m56.623164793s) [upload=544685]: Upload finished. err: <nil>
Duration: 30.004727937s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
12.822204536s trace:
https://jaeger.dev.storj.io/trace/51ca50ba1c5675ff
FAIL (2025-12-31 13:04:17 UTC)
orbiter warning: context deadline exceeded
Duration: 12.822237249s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............X.................X...........................] duration:
15.694307267s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:03:50 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 15.69433043s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
2.455626496s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:03:47 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 2.455653036s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [...X.X........................X.............................] duration:
3.778379819s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:03:42 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.77844322s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X.XXXX..X...X.......X......X.X...X.X........XX..X.X..X....X] duration:
30.003938747s trace:
https://jaeger.dev.storj.io/trace/1470758fad619452
FAIL (2025-12-31 13:03:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:03:06.684645346 (621h34m6.555728391s) [upload=544670]: Starting upload
Dec 31 13:03:06.684748652 (621h34m6.555831788s) [upload=544670]: Done starting upload
Dec 31 13:03:06.684824458 (621h34m6.555907493s) [upload=544670]: Uploading segments...
Dec 31 13:03:06.685025910 (621h34m6.556108955s) [upload=544670]: Got next segment. Inline: false
Dec 31 13:03:06.685062059 (621h34m6.556145094s) [upload=544670]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:03:06.685066578 (621h34m6.556149613s) [upload=544670]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:03:06.930052357 (621h34m6.801135402s) [seg_pos={0 0}, upload=544670]: Begin upload segment...
Dec 31 13:03:06.930118002 (621h34m6.801201037s) [seg_pos={0 0}, upload=544670]: Done begin upload segment.
Dec 31 13:03:06.930176785 (621h34m6.80125983s) [noise=true , address=145.79.227.2:4113 , piece_id=ELL4OK5GSUAWMHVQQ5VROQLYFGSO5ELWHS3LFRMS7XZLE3BKCJ4Q, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.930211360 (621h34m6.801294396s) [noise=true , address=145.79.227.43:4158 , piece_id=F3BZLH7H5RAMPEESOVPBPIIE3LJ4WHPGUBRCG5D675ACGJIOQG5Q, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.930941801 (621h34m6.802024846s) [noise=true , address=145.79.227.36:4519 , piece_id=H4YJMXCPOIXJBG4WVKAPNQVUT7MSUV6P3RXOUJOPDYNTWD23NLJA, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.931100463 (621h34m6.802183508s) [noise=true , address=77.90.31.14:4662 , piece_id=EVSEJZLQOPGLYH2H6K6EMHBSMT6VBTO33YAQ42EQD7OUDLMP3RSA, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.931626024 (621h34m6.802709069s) [noise=true , address=77.90.31.21:4877 , piece_id=2VUFS6QNMCWLOFJV6ALA5TVCD4STIP5BJWSKW3BMUG4VUCRIH7EQ, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.932072414 (621h34m6.803155449s) [noise=true , address=45.91.116.7:4010 , piece_id=XUZ6POXLAVA7KHCINP7TDPNRI5XFYGCXTV7P4R2TFSELXUFKVHFA, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.932688237 (621h34m6.803771282s) [noise=true , address=145.79.227.12:4765 , piece_id=MFNRZ7L3YR3RK5YXLKRU3M5UZJEZSEIKHG54T3C4HDBJAS5A2IRQ, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.933515733 (621h34m6.804598778s) [noise=true , address=145.79.227.31:4798 , piece_id=5BPPXALMMF5JXK36H6ZAMWPSQSQMAFRUM46N6UW4S3VFAYUSCA3Q, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.934604976 (621h34m6.805688012s) [noise=true , address=145.79.227.25:4619 , piece_id=EK7XQXVDUY7KDE4XRUMLPMPJYQ2NWFQQZY36Y63NBYL4PBLMJ6IA, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.935145315 (621h34m6.80622836s) [noise=true , address=145.79.227.52:4445 , piece_id=ZZCDTM6E2LTRLP5RKCAM3OW3NHWZN4J3Q3KTU42GPKUDXDKW644Q, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.935626050 (621h34m6.806709085s) [noise=true , address=45.91.116.14:4189 , piece_id=XMD4NJEMR67GP4SYM2GBAPPRXVDANNWGBI6O2Y3TFQ5FHE4MKIWA, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.936104021 (621h34m6.807187056s) [noise=true , address=145.79.227.47:4267 , piece_id=5XM7L3DBARMEUWGUN4PT77UNDSXLKKSFYUKF3HKAGKQLXIMOYZOA, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.936593533 (621h34m6.807676578s) [noise=true , address=145.79.227.50:4375 , piece_id=YM5H56LVT5YHPD5ZYTD67PRWEDEO2H7M6DEFVO2UT4TUTCMJLVDA, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.937266033 (621h34m6.808349068s) [noise=true , address=77.90.31.10:4517 , piece_id=YW4CKKFCHUSPY62Z4PGAMVIZHAGA42S562VWXENX4BILRHEPBB5A, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.937961758 (621h34m6.809044793s) [noise=true , address=145.79.227.45:4231 , piece_id=O6DTIZPEGY5TJT7NJF5YOHJ2U2Q7XPGP2IXPVWYEMNI4D432GADQ, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:06.938130600 (621h34m6.809213655s) [noise=true , address=145.79.227.39:4013 , piece_id=DA3UE4CF4QP336CWCXYW6KY4ZOTDFGQV72OAWSDMC6JLMAGKABNA, seg_pos={0 0}, upload=544670]: Uploading piece...
Dec 31 13:03:08.608542863 (621h34m8.479625899s) [upload=544670]: Next returned nil segment
Dec 31 13:03:08.608555448 (621h34m8.479638483s) [upload=544670]: Waiting for error group managing segments...
Dec 31 13:03:08.901881904 (621h34m8.772964939s) [noise=true , address=145.79.227.2:4113 , piece_id=ELL4OK5GSUAWMHVQQ5VROQLYFGSO5ELWHS3LFRMS7XZLE3BKCJ4Q, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:08.922578704 (621h34m8.793661739s) [noise=true , address=145.79.227.36:4519 , piece_id=H4YJMXCPOIXJBG4WVKAPNQVUT7MSUV6P3RXOUJOPDYNTWD23NLJA, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:08.928124312 (621h34m8.799207357s) [noise=true , address=145.79.227.52:4445 , piece_id=ZZCDTM6E2LTRLP5RKCAM3OW3NHWZN4J3Q3KTU42GPKUDXDKW644Q, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:08.928175921 (621h34m8.799258956s) [noise=true , address=145.79.227.25:4619 , piece_id=EK7XQXVDUY7KDE4XRUMLPMPJYQ2NWFQQZY36Y63NBYL4PBLMJ6IA, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:08.930842008 (621h34m8.801925043s) [noise=true , address=145.79.227.45:4231 , piece_id=O6DTIZPEGY5TJT7NJF5YOHJ2U2Q7XPGP2IXPVWYEMNI4D432GADQ, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:08.938794240 (621h34m8.809877285s) [noise=true , address=145.79.227.31:4798 , piece_id=5BPPXALMMF5JXK36H6ZAMWPSQSQMAFRUM46N6UW4S3VFAYUSCA3Q, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:08.946186416 (621h34m8.817269461s) [noise=true , address=145.79.227.50:4375 , piece_id=YM5H56LVT5YHPD5ZYTD67PRWEDEO2H7M6DEFVO2UT4TUTCMJLVDA, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:08.952881444 (621h34m8.823964489s) [noise=true , address=145.79.227.39:4013 , piece_id=DA3UE4CF4QP336CWCXYW6KY4ZOTDFGQV72OAWSDMC6JLMAGKABNA, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:09.006024265 (621h34m8.8771073s) [noise=true , address=145.79.227.12:4765 , piece_id=MFNRZ7L3YR3RK5YXLKRU3M5UZJEZSEIKHG54T3C4HDBJAS5A2IRQ, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:09.019841675 (621h34m8.89092472s) [noise=true , address=145.79.227.47:4267 , piece_id=5XM7L3DBARMEUWGUN4PT77UNDSXLKKSFYUKF3HKAGKQLXIMOYZOA, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:09.110396696 (621h34m8.981479731s) [noise=true , address=77.90.31.14:4662 , piece_id=EVSEJZLQOPGLYH2H6K6EMHBSMT6VBTO33YAQ42EQD7OUDLMP3RSA, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:09.118428429 (621h34m8.989511464s) [noise=true , address=77.90.31.21:4877 , piece_id=2VUFS6QNMCWLOFJV6ALA5TVCD4STIP5BJWSKW3BMUG4VUCRIH7EQ, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:09.305090421 (621h34m9.176173466s) [noise=true , address=45.91.116.7:4010 , piece_id=XUZ6POXLAVA7KHCINP7TDPNRI5XFYGCXTV7P4R2TFSELXUFKVHFA, seg_pos={0 0}, upload=544670]: Done uploading piece. err: <nil>
Dec 31 13:03:09.305115337 (621h34m9.176198373s) [seg_pos={0 0}, upload=544670]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:03:09.305334696 (621h34m9.176417741s) [noise=true , address=77.90.31.10:4517 , piece_id=YW4CKKFCHUSPY62Z4PGAMVIZHAGA42S562VWXENX4BILRHEPBB5A, seg_pos={0 0}, upload=544670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:09.305343462 (621h34m9.176426507s) [noise=true , address=77.90.31.10:4517 , piece_id=YW4CKKFCHUSPY62Z4PGAMVIZHAGA42S562VWXENX4BILRHEPBB5A, seg_pos={0 0}, upload=544670]: Long tail cancellation due to optimal threshold
Dec 31 13:03:09.305722164 (621h34m9.176805209s) [noise=true , address=45.91.116.14:4189 , piece_id=XMD4NJEMR67GP4SYM2GBAPPRXVDANNWGBI6O2Y3TFQ5FHE4MKIWA, seg_pos={0 0}, upload=544670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:09.305735749 (621h34m9.176818794s) [noise=true , address=45.91.116.14:4189 , piece_id=XMD4NJEMR67GP4SYM2GBAPPRXVDANNWGBI6O2Y3TFQ5FHE4MKIWA, seg_pos={0 0}, upload=544670]: Long tail cancellation due to optimal threshold
Dec 31 13:03:09.305757972 (621h34m9.176841007s) [noise=true , address=145.79.227.43:4158 , piece_id=F3BZLH7H5RAMPEESOVPBPIIE3LJ4WHPGUBRCG5D675ACGJIOQG5Q, seg_pos={0 0}, upload=544670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 31 13:03:09.305768031 (621h34m9.176851066s) [noise=true , address=145.79.227.43:4158 , piece_id=F3BZLH7H5RAMPEESOVPBPIIE3LJ4WHPGUBRCG5D675ACGJIOQG5Q, seg_pos={0 0}, upload=544670]: Long tail cancellation due to optimal threshold
Dec 31 13:03:09.305787949 (621h34m9.176870984s) [seg_pos={0 0}, upload=544670]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 13:03:09.305971578 (621h34m9.177054623s) [upload=544670]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:03:09.305978712 (621h34m9.177061757s) [upload=544670]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:03:09.604290942 (621h34m9.475373977s) [upload=544670]: Done uploading segments...
Dec 31 13:03:09.604310349 (621h34m9.475393394s) [upload=544670]: Upload finished. err: <nil>
Duration: 30.003961921s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............X.................X..........................] duration:
32.252262361s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:03:22 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 32.252291878s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.42141619s trace:
https://jaeger.dev.storj.io/trace/20e918ed48f6bce2
FAIL (2025-12-31 13:03:14 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.421450584s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [....X.X........................X............................] duration:
3.549588066s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:03:03 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.549677887s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
12.255280829s trace:
https://jaeger.dev.storj.io/trace/51ca50ba1c5675ff
FAIL (2025-12-31 13:03:02 UTC)
orbiter warning: context deadline exceeded
Duration: 12.255310728s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..X.XXXX..X...X.......X......X.X...X.X........XX..X.X..X....] duration:
6.10716382s trace:
https://jaeger.dev.storj.io/trace/2b47efb9dfa90aa5
FAIL (2025-12-31 13:02:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:00:04.829321691 (621h31m4.700404746s) [upload=544628]: Starting upload
Dec 31 13:00:04.829400642 (621h31m4.700483687s) [upload=544628]: Done starting upload
Dec 31 13:00:04.829745378 (621h31m4.700828423s) [upload=544628]: Uploading segments...
Dec 31 13:00:04.829983992 (621h31m4.701067037s) [upload=544628]: Got next segment. Inline: false
Dec 31 13:00:04.830036622 (621h31m4.701119677s) [upload=544628]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:00:04.830077420 (621h31m4.701160475s) [upload=544628]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:00:05.077557476 (621h31m4.948640521s) [seg_pos={0 0}, upload=544628]: Begin upload segment...
Dec 31 13:00:05.077620206 (621h31m4.948703241s) [seg_pos={0 0}, upload=544628]: Done begin upload segment.
Dec 31 13:00:05.077656986 (621h31m4.948740041s) [noise=true , address=145.79.227.20:4438 , piece_id=USYQDKAAEMBWBSLXM2LXGK42QHELP7O7XZU7ELYMRS3MICBOUXXA, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.080101460 (621h31m4.951184505s) [noise=true , address=45.91.116.13:4155 , piece_id=QL44OW6LGFACBNVSI5GJBXYHYPE2ELZIBZHWIV65VT5QDMBPGF7A, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.081374345 (621h31m4.95245739s) [noise=true , address=145.79.227.7:4295 , piece_id=VEFG6Y6D6ATPBNY53RDW65SBOEKO4GP6F7KCN4V5UYBQTKCPJA5Q, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.081949499 (621h31m4.953032544s) [noise=true , address=45.91.116.12:4228 , piece_id=U3ZXJYBIY4WAXZCWCS7GVPS74OWE7POKDWVGO6I7OF2NGGONCQQQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.082300357 (621h31m4.953383402s) [noise=true , address=145.79.227.6:4258 , piece_id=BENH7HBVOFPOIGIH4H3K7R6J5PDWTCUFSMNMND5FJTBVYFJ6FGNQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.082527410 (621h31m4.953610455s) [noise=true , address=145.79.227.24:4582 , piece_id=JN4MMPODZN3LNZY3EBMIZFQ4W7GGGNHN7UTVNJXJMKXQE4I53YSQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.082962859 (621h31m4.954045904s) [noise=true , address=145.79.227.48:4303 , piece_id=DU5CG7XUFFWHINPGV6BIF4QELPAV63OSQ2ROE3674UMXFDNRCGNQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.082988408 (621h31m4.954071453s) [noise=true , address=145.79.227.33:4870 , piece_id=OLBGTW3FM3GIINWKLRAX4VLTCULA3ANLL5H55SBVY2UVVE6PX7CA, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.083652502 (621h31m4.954735557s) [noise=true , address=45.91.116.14:4191 , piece_id=WIGTUMGW6PY7XAEDM6YZTZYYZKXFADZKDOZO7ATW2FKUGWU6XJNQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.083806236 (621h31m4.954889281s) [noise=true , address=145.79.227.40:4050 , piece_id=NUYBMSM6RGCYBKVKTVA5GT5E22XUBFGPIGUCGFMORTR3LL3GIFBA, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.084141774 (621h31m4.955224819s) [noise=true , address=77.90.31.21:4877 , piece_id=NVDSV5EALWSH7JMRV3OP7GAIRDRIAZDU2PW6ZWIDSSZB43F5OWIQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.084374668 (621h31m4.955457713s) [noise=true , address=45.91.116.8:4045 , piece_id=QBYICQH6UJYQE42XFQLXDXIAOYNKXHDLRWSHZ7F2L4I2IZUJIZUQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.084680089 (621h31m4.955763134s) [noise=true , address=145.79.227.45:4230 , piece_id=RCVTATPOW2XDBPS6EDBDVHXER4NXAE7F5KOIL5GDWQXHGC2LSJUA, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.085366375 (621h31m4.956449421s) [noise=true , address=145.79.227.51:4410 , piece_id=LENMIT7BSYXMTZOGYVIWYLGZKI6Z33MFTOZIZXPK4KRVNU4PFL3A, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.086150439 (621h31m4.957233484s) [noise=true , address=145.79.227.11:4729 , piece_id=74TYI7M2WF3DFTURN5DIWVIIUTJFKTQF4Y4ONCWYM6GFNFDTSEYQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.086848007 (621h31m4.957931052s) [noise=true , address=77.90.31.16:4735 , piece_id=63ARO6EOFP3JAOUXB73I4KY6KKO5U2EXG5MUHJ36GHCFS3NPR3RA, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:06.709092104 (621h31m6.580175149s) [upload=544628]: Next returned nil segment
Dec 31 13:00:06.709126049 (621h31m6.580209114s) [upload=544628]: Waiting for error group managing segments...
Dec 31 13:00:07.079272891 (621h31m6.950355936s) [noise=true , address=145.79.227.45:4230 , piece_id=RCVTATPOW2XDBPS6EDBDVHXER4NXAE7F5KOIL5GDWQXHGC2LSJUA, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.093831062 (621h31m6.964914097s) [noise=true , address=145.79.227.20:4438 , piece_id=USYQDKAAEMBWBSLXM2LXGK42QHELP7O7XZU7ELYMRS3MICBOUXXA, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.096629902 (621h31m6.967712938s) [noise=true , address=145.79.227.6:4258 , piece_id=BENH7HBVOFPOIGIH4H3K7R6J5PDWTCUFSMNMND5FJTBVYFJ6FGNQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.097659623 (621h31m6.968742668s) [noise=true , address=145.79.227.48:4303 , piece_id=DU5CG7XUFFWHINPGV6BIF4QELPAV63OSQ2ROE3674UMXFDNRCGNQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.100386274 (621h31m6.971469319s) [noise=true , address=145.79.227.24:4582 , piece_id=JN4MMPODZN3LNZY3EBMIZFQ4W7GGGNHN7UTVNJXJMKXQE4I53YSQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.101142925 (621h31m6.97222597s) [noise=true , address=145.79.227.7:4295 , piece_id=VEFG6Y6D6ATPBNY53RDW65SBOEKO4GP6F7KCN4V5UYBQTKCPJA5Q, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.109980364 (621h31m6.981063399s) [noise=true , address=145.79.227.11:4729 , piece_id=74TYI7M2WF3DFTURN5DIWVIIUTJFKTQF4Y4ONCWYM6GFNFDTSEYQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.194475074 (621h31m7.065558119s) [noise=true , address=145.79.227.51:4410 , piece_id=LENMIT7BSYXMTZOGYVIWYLGZKI6Z33MFTOZIZXPK4KRVNU4PFL3A, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.248642285 (621h31m7.119725331s) [noise=true , address=77.90.31.16:4735 , piece_id=63ARO6EOFP3JAOUXB73I4KY6KKO5U2EXG5MUHJ36GHCFS3NPR3RA, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.284496677 (621h31m7.155579723s) [noise=true , address=77.90.31.21:4877 , piece_id=NVDSV5EALWSH7JMRV3OP7GAIRDRIAZDU2PW6ZWIDSSZB43F5OWIQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.443064013 (621h31m7.314147048s) [noise=true , address=45.91.116.14:4191 , piece_id=WIGTUMGW6PY7XAEDM6YZTZYYZKXFADZKDOZO7ATW2FKUGWU6XJNQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.492591942 (621h31m7.363674987s) [noise=true , address=45.91.116.13:4155 , piece_id=QL44OW6LGFACBNVSI5GJBXYHYPE2ELZIBZHWIV65VT5QDMBPGF7A, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.494736765 (621h31m7.36581982s) [noise=true , address=145.79.227.33:4870 , piece_id=OLBGTW3FM3GIINWKLRAX4VLTCULA3ANLL5H55SBVY2UVVE6PX7CA, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.494762424 (621h31m7.36584547s) [seg_pos={0 0}, upload=544628]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:00:07.495072265 (621h31m7.36615531s) [noise=true , address=145.79.227.40:4050 , piece_id=NUYBMSM6RGCYBKVKTVA5GT5E22XUBFGPIGUCGFMORTR3LL3GIFBA, seg_pos={0 0}, upload=544628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 31 13:00:07.495081522 (621h31m7.366164577s) [noise=true , address=145.79.227.40:4050 , piece_id=NUYBMSM6RGCYBKVKTVA5GT5E22XUBFGPIGUCGFMORTR3LL3GIFBA, seg_pos={0 0}, upload=544628]: Long tail cancellation due to optimal threshold
Dec 31 13:00:07.495142198 (621h31m7.366225253s) [noise=true , address=45.91.116.12:4228 , piece_id=U3ZXJYBIY4WAXZCWCS7GVPS74OWE7POKDWVGO6I7OF2NGGONCQQQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.496067199 (621h31m7.367150254s) [noise=true , address=45.91.116.8:4045 , piece_id=QBYICQH6UJYQE42XFQLXDXIAOYNKXHDLRWSHZ7F2L4I2IZUJIZUQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 31 13:00:07.496079812 (621h31m7.367162857s) [noise=true , address=45.91.116.8:4045 , piece_id=QBYICQH6UJYQE42XFQLXDXIAOYNKXHDLRWSHZ7F2L4I2IZUJIZUQ, seg_pos={0 0}, upload=544628]: Long tail cancellation due to optimal threshold
Dec 31 13:00:07.496140778 (621h31m7.367223824s) [seg_pos={0 0}, upload=544628]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 31 13:00:07.496330110 (621h31m7.367413155s) [upload=544628]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:00:07.496338976 (621h31m7.367422021s) [upload=544628]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:00:07.789079667 (621h31m7.660162732s) [upload=544628]: Done uploading segments...
Dec 31 13:00:07.789112900 (621h31m7.660195935s) [upload=544628]: Upload finished. err: <nil>
Duration: 6.107189139s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
8.539393279s trace:
https://jaeger.dev.storj.io/trace/51ca50ba1c5675ff
FAIL (2025-12-31 13:02:37 UTC)
orbiter warning: context deadline exceeded
Duration: 8.539417916s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [.....X.X........................X...........................] duration:
3.917677182s trace:
https://jaeger.dev.storj.io/trace/32a738dc9de3e27a
FAIL (2025-12-31 13:02:37 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.917702201s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [...X.XXXX..X...X.......X......X.X...X.X........XX..X.X..X...] duration:
5.985876083s trace:
https://jaeger.dev.storj.io/trace/2b47efb9dfa90aa5
FAIL (2025-12-31 13:02:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 13:00:04.829321691 (621h31m4.700404746s) [upload=544628]: Starting upload
Dec 31 13:00:04.829400642 (621h31m4.700483687s) [upload=544628]: Done starting upload
Dec 31 13:00:04.829745378 (621h31m4.700828423s) [upload=544628]: Uploading segments...
Dec 31 13:00:04.829983992 (621h31m4.701067037s) [upload=544628]: Got next segment. Inline: false
Dec 31 13:00:04.830036622 (621h31m4.701119677s) [upload=544628]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 13:00:04.830077420 (621h31m4.701160475s) [upload=544628]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 13:00:05.077557476 (621h31m4.948640521s) [seg_pos={0 0}, upload=544628]: Begin upload segment...
Dec 31 13:00:05.077620206 (621h31m4.948703241s) [seg_pos={0 0}, upload=544628]: Done begin upload segment.
Dec 31 13:00:05.077656986 (621h31m4.948740041s) [noise=true , address=145.79.227.20:4438 , piece_id=USYQDKAAEMBWBSLXM2LXGK42QHELP7O7XZU7ELYMRS3MICBOUXXA, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.080101460 (621h31m4.951184505s) [noise=true , address=45.91.116.13:4155 , piece_id=QL44OW6LGFACBNVSI5GJBXYHYPE2ELZIBZHWIV65VT5QDMBPGF7A, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.081374345 (621h31m4.95245739s) [noise=true , address=145.79.227.7:4295 , piece_id=VEFG6Y6D6ATPBNY53RDW65SBOEKO4GP6F7KCN4V5UYBQTKCPJA5Q, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.081949499 (621h31m4.953032544s) [noise=true , address=45.91.116.12:4228 , piece_id=U3ZXJYBIY4WAXZCWCS7GVPS74OWE7POKDWVGO6I7OF2NGGONCQQQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.082300357 (621h31m4.953383402s) [noise=true , address=145.79.227.6:4258 , piece_id=BENH7HBVOFPOIGIH4H3K7R6J5PDWTCUFSMNMND5FJTBVYFJ6FGNQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.082527410 (621h31m4.953610455s) [noise=true , address=145.79.227.24:4582 , piece_id=JN4MMPODZN3LNZY3EBMIZFQ4W7GGGNHN7UTVNJXJMKXQE4I53YSQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.082962859 (621h31m4.954045904s) [noise=true , address=145.79.227.48:4303 , piece_id=DU5CG7XUFFWHINPGV6BIF4QELPAV63OSQ2ROE3674UMXFDNRCGNQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.082988408 (621h31m4.954071453s) [noise=true , address=145.79.227.33:4870 , piece_id=OLBGTW3FM3GIINWKLRAX4VLTCULA3ANLL5H55SBVY2UVVE6PX7CA, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.083652502 (621h31m4.954735557s) [noise=true , address=45.91.116.14:4191 , piece_id=WIGTUMGW6PY7XAEDM6YZTZYYZKXFADZKDOZO7ATW2FKUGWU6XJNQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.083806236 (621h31m4.954889281s) [noise=true , address=145.79.227.40:4050 , piece_id=NUYBMSM6RGCYBKVKTVA5GT5E22XUBFGPIGUCGFMORTR3LL3GIFBA, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.084141774 (621h31m4.955224819s) [noise=true , address=77.90.31.21:4877 , piece_id=NVDSV5EALWSH7JMRV3OP7GAIRDRIAZDU2PW6ZWIDSSZB43F5OWIQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.084374668 (621h31m4.955457713s) [noise=true , address=45.91.116.8:4045 , piece_id=QBYICQH6UJYQE42XFQLXDXIAOYNKXHDLRWSHZ7F2L4I2IZUJIZUQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.084680089 (621h31m4.955763134s) [noise=true , address=145.79.227.45:4230 , piece_id=RCVTATPOW2XDBPS6EDBDVHXER4NXAE7F5KOIL5GDWQXHGC2LSJUA, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.085366375 (621h31m4.956449421s) [noise=true , address=145.79.227.51:4410 , piece_id=LENMIT7BSYXMTZOGYVIWYLGZKI6Z33MFTOZIZXPK4KRVNU4PFL3A, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.086150439 (621h31m4.957233484s) [noise=true , address=145.79.227.11:4729 , piece_id=74TYI7M2WF3DFTURN5DIWVIIUTJFKTQF4Y4ONCWYM6GFNFDTSEYQ, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:05.086848007 (621h31m4.957931052s) [noise=true , address=77.90.31.16:4735 , piece_id=63ARO6EOFP3JAOUXB73I4KY6KKO5U2EXG5MUHJ36GHCFS3NPR3RA, seg_pos={0 0}, upload=544628]: Uploading piece...
Dec 31 13:00:06.709092104 (621h31m6.580175149s) [upload=544628]: Next returned nil segment
Dec 31 13:00:06.709126049 (621h31m6.580209114s) [upload=544628]: Waiting for error group managing segments...
Dec 31 13:00:07.079272891 (621h31m6.950355936s) [noise=true , address=145.79.227.45:4230 , piece_id=RCVTATPOW2XDBPS6EDBDVHXER4NXAE7F5KOIL5GDWQXHGC2LSJUA, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.093831062 (621h31m6.964914097s) [noise=true , address=145.79.227.20:4438 , piece_id=USYQDKAAEMBWBSLXM2LXGK42QHELP7O7XZU7ELYMRS3MICBOUXXA, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.096629902 (621h31m6.967712938s) [noise=true , address=145.79.227.6:4258 , piece_id=BENH7HBVOFPOIGIH4H3K7R6J5PDWTCUFSMNMND5FJTBVYFJ6FGNQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.097659623 (621h31m6.968742668s) [noise=true , address=145.79.227.48:4303 , piece_id=DU5CG7XUFFWHINPGV6BIF4QELPAV63OSQ2ROE3674UMXFDNRCGNQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.100386274 (621h31m6.971469319s) [noise=true , address=145.79.227.24:4582 , piece_id=JN4MMPODZN3LNZY3EBMIZFQ4W7GGGNHN7UTVNJXJMKXQE4I53YSQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.101142925 (621h31m6.97222597s) [noise=true , address=145.79.227.7:4295 , piece_id=VEFG6Y6D6ATPBNY53RDW65SBOEKO4GP6F7KCN4V5UYBQTKCPJA5Q, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.109980364 (621h31m6.981063399s) [noise=true , address=145.79.227.11:4729 , piece_id=74TYI7M2WF3DFTURN5DIWVIIUTJFKTQF4Y4ONCWYM6GFNFDTSEYQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.194475074 (621h31m7.065558119s) [noise=true , address=145.79.227.51:4410 , piece_id=LENMIT7BSYXMTZOGYVIWYLGZKI6Z33MFTOZIZXPK4KRVNU4PFL3A, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.248642285 (621h31m7.119725331s) [noise=true , address=77.90.31.16:4735 , piece_id=63ARO6EOFP3JAOUXB73I4KY6KKO5U2EXG5MUHJ36GHCFS3NPR3RA, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.284496677 (621h31m7.155579723s) [noise=true , address=77.90.31.21:4877 , piece_id=NVDSV5EALWSH7JMRV3OP7GAIRDRIAZDU2PW6ZWIDSSZB43F5OWIQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.443064013 (621h31m7.314147048s) [noise=true , address=45.91.116.14:4191 , piece_id=WIGTUMGW6PY7XAEDM6YZTZYYZKXFADZKDOZO7ATW2FKUGWU6XJNQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.492591942 (621h31m7.363674987s) [noise=true , address=45.91.116.13:4155 , piece_id=QL44OW6LGFACBNVSI5GJBXYHYPE2ELZIBZHWIV65VT5QDMBPGF7A, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.494736765 (621h31m7.36581982s) [noise=true , address=145.79.227.33:4870 , piece_id=OLBGTW3FM3GIINWKLRAX4VLTCULA3ANLL5H55SBVY2UVVE6PX7CA, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.494762424 (621h31m7.36584547s) [seg_pos={0 0}, upload=544628]: Segment reached optimal threshold of 13 pieces.
Dec 31 13:00:07.495072265 (621h31m7.36615531s) [noise=true , address=145.79.227.40:4050 , piece_id=NUYBMSM6RGCYBKVKTVA5GT5E22XUBFGPIGUCGFMORTR3LL3GIFBA, seg_pos={0 0}, upload=544628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 31 13:00:07.495081522 (621h31m7.366164577s) [noise=true , address=145.79.227.40:4050 , piece_id=NUYBMSM6RGCYBKVKTVA5GT5E22XUBFGPIGUCGFMORTR3LL3GIFBA, seg_pos={0 0}, upload=544628]: Long tail cancellation due to optimal threshold
Dec 31 13:00:07.495142198 (621h31m7.366225253s) [noise=true , address=45.91.116.12:4228 , piece_id=U3ZXJYBIY4WAXZCWCS7GVPS74OWE7POKDWVGO6I7OF2NGGONCQQQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: <nil>
Dec 31 13:00:07.496067199 (621h31m7.367150254s) [noise=true , address=45.91.116.8:4045 , piece_id=QBYICQH6UJYQE42XFQLXDXIAOYNKXHDLRWSHZ7F2L4I2IZUJIZUQ, seg_pos={0 0}, upload=544628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF
Dec 31 13:00:07.496079812 (621h31m7.367162857s) [noise=true , address=45.91.116.8:4045 , piece_id=QBYICQH6UJYQE42XFQLXDXIAOYNKXHDLRWSHZ7F2L4I2IZUJIZUQ, seg_pos={0 0}, upload=544628]: Long tail cancellation due to optimal threshold
Dec 31 13:00:07.496140778 (621h31m7.367223824s) [seg_pos={0 0}, upload=544628]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 31 13:00:07.496330110 (621h31m7.367413155s) [upload=544628]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 13:00:07.496338976 (621h31m7.367422021s) [upload=544628]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 13:00:07.789079667 (621h31m7.660162732s) [upload=544628]: Done uploading segments...
Dec 31 13:00:07.789112900 (621h31m7.660195935s) [upload=544628]: Upload finished. err: <nil>
Duration: 5.985904308s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [................X.................X.........................] duration:
30.706400786s trace:
https://jaeger.dev.storj.io/trace/e9269665ab7d695
FAIL (2025-12-31 13:02:31 UTC)
orbiter warning:
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
1h13m9.490069934s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619022437 (621h13m9.490105482s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619621908 (621h13m9.490704953s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.619977346 (621h13m9.491060381s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620122993 (621h13m9.491206038s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620606965 (621h13m9.49169002s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.620733515 (621h13m9.4918166s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621087228 (621h13m9.492170274s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621509342 (621h13m9.492592387s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.621583985 (621h13m9.49266703s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622118322 (621h13m9.493201367s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622349181 (621h13m9.493432226s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.622668250 (621h13m9.493751295s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.623177278 (621h13m9.494260313s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624038138 (621h13m9.495121183s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:09.624286150 (621h13m9.495369205s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Uploading piece...
Dec 31 12:42:11.071488329 (621h13m10.942571374s) [upload=544367]: Next returned nil segment
Dec 31 12:42:11.071505553 (621h13m10.942588588s) [upload=544367]: Waiting for error group managing segments...
Dec 31 12:42:11.493782215 (621h13m11.36486526s) [noise=true , address=145.79.227.34:4932 , piece_id=AUAUI6NCS6KDOCCBTUL7JG4R452WQZK6L7PT6GCC4Z7DNLRCMTNA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.518424607 (621h13m11.389507652s) [noise=true , address=64.49.13.26:4949 , piece_id=IYDO6JG4P4SGBCKXJBD3FKAJZLWLJOL6JSXW7EBVXYOTWNGBRCOA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.574160737 (621h13m11.445243782s) [noise=true , address=145.79.227.4:4212 , piece_id=BEHTANTZT2A5VX74EOO7CYBNLKXBVBOSFRTQCALKNJUI76UAIOQQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.657253546 (621h13m11.528336581s) [noise=true , address=77.90.31.19:4848 , piece_id=E7C6GKW3G3M5SWUG4IKHCKCRJEDTQZOEXYHFRKL3AIGQYZILTLWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.675374742 (621h13m11.546457777s) [noise=true , address=145.79.227.29:4973 , piece_id=ARHQUOCSMVRBLSKTAY2TADQKVMJGTSSYUE4GM6EFUUCFRERWMGCQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.722110212 (621h13m11.593193258s) [noise=true , address=77.90.31.12:4599 , piece_id=6OPSMNBR3EUN2KZQQ2CYQM64TWC6OPYWDSORMFOEWFKN7AOGOAKA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.840586577 (621h13m11.711669623s) [noise=true , address=77.90.31.11:4581 , piece_id=IFKIZVY3QJRVNVUNKKHOO45XRQAWJDITVVWE3QLSRMPWUHYJRDHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:11.914711151 (621h13m11.785794186s) [noise=true , address=45.91.116.6:4003 , piece_id=BF5E72OZJYQP3E42AOQ6O3JJOS3RYSHN4FCQPL7Y3JUXAQQHEWDA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.316417195 (621h13m12.18750024s) [noise=true , address=45.91.116.14:4203 , piece_id=YAWCOTBYE4AFFDMOO3IYRGNNEKIIBM7R5ME7TT3G7N7IZRP7Q2LA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.386327474 (621h13m12.257410519s) [noise=true , address=77.90.31.17:4803 , piece_id=GBE4WXW2R2NK6W77O7TZWOYYI5EHBIXEBQ7BDET4JJQL2HXG2FEA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.412099617 (621h13m12.283182662s) [noise=true , address=145.79.227.30:4773 , piece_id=JNQYWQ3PCBBN56YFAS5LPTRC4QDF7L36TYVPP7VIXAO37TOVQZIQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.437429018 (621h13m12.308512053s) [noise=true , address=145.79.227.44:4204 , piece_id=BZJUFGCIHK3EDTGLFNG7OIDGZBAU4K4IYWV734WHA6LMLBKKCTHA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.438056774 (621h13m12.309139809s) [noise=true , address=77.90.31.16:4745 , piece_id=2UVWCPJS2QXYIONERKNLGAXYMXGJTG74JW2LYGB45I66VS2JPBLQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455032857 (621h13m12.326115892s) [noise=true , address=145.79.227.43:4171 , piece_id=PR7ATPRO6ACYLYWQ7ENRETZZNL5XPIOI5NMVELYERH23R7ABJJIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: <nil>
Dec 31 12:42:12.455053367 (621h13m12.326136402s) [seg_pos={0 0}, upload=544367]: Segment reached optimal threshold of 14 pieces.
Dec 31 12:42:12.455345723 (621h13m12.326428758s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK): protocol: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45372->64.49.13.169:4715: use of closed network connection
Dec 31 12:42:12.455352437 (621h13m12.326435482s) [noise=true , address=64.49.13.169:4715 , piece_id=WNLGFRGLAM6E6WAHPSPUI6KHPHO33ATI52RWPX3LSI7LG354OTWQ, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455417200 (621h13m12.326500235s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455468046 (621h13m12.326551091s) [noise=true , address=145.79.227.33:4880 , piece_id=R7YKQXDXQEXJEF7U5CVMGBLEKJ53ME3HK37WBOZDNZDY3LDS4FJA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455487414 (621h13m12.326570459s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455493465 (621h13m12.3265765s) [noise=true , address=145.79.227.49:4340 , piece_id=5CPZ5EACX2Z3B7A6C23E5Q27KVB3E57CH3J2TRYQLLZBFJ35A6KA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455605057 (621h13m12.326688102s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY): protocol: expected piece hash; context canceled; EOF
Dec 31 12:42:12.455611139 (621h13m12.326694194s) [noise=true , address=145.79.227.40:4059 , piece_id=GBJGU2FM65CUWI2DH2NDKXYO3ZTLVAJVXLKX4I232EYTGYHEEUIA, seg_pos={0 0}, upload=544367]: Long tail cancellation due to optimal threshold
Dec 31 12:42:12.455729374 (621h13m12.326812409s) [seg_pos={0 0}, upload=544367]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 31 12:42:12.455800239 (621h13m12.326883274s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 12:42:12.455806241 (621h13m12.326889276s) [upload=544367]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 12:42:12.770280101 (621h13m12.641363146s) [upload=544367]: Done uploading segments...
Dec 31 12:42:12.770300369 (621h13m12.641383404s) [upload=544367]: Upload finished. err: <nil>
Duration: 30.706425592s
core
uplink
us-select-1
us1-placement-20
|