Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.....X....X...................................] duration: 5.768929616s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:28:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.76895266s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001321956s trace: https://jaeger.dev.storj.io/trace/684278858089df61 FAIL (2026-01-09 07:27:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001377362s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 15.967661173s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:27:41 UTC) orbiter warning: context deadline exceeded Duration: 15.9676862s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.............X.....X....X..................................] duration: 5.703842543s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:27:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.703868882s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001774841s trace: https://jaeger.dev.storj.io/trace/7537908e58ed74a5 FAIL (2026-01-09 07:26:45 UTC) orbiter warning: context deadline exceeded Duration: 5.001796621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 7.245409585s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:26:43 UTC) orbiter warning: context deadline exceeded Duration: 7.245438221s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.............X.....X....X.................................] duration: 5.735274986s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:26:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.735304041s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 10.337809567s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:25:59 UTC) orbiter warning: context deadline exceeded Duration: 10.337834224s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration: 5.001364658s trace: https://jaeger.dev.storj.io/trace/5e49126d9d09ac58 FAIL (2026-01-09 07:25:44 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001386589s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 12.230965211s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:25:21 UTC) orbiter warning: context deadline exceeded Duration: 12.231000918s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X.............X.....X....X................................] duration: 5.842718682s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:25:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.842750433s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration: 5.001005553s trace: https://jaeger.dev.storj.io/trace/3d926781ac8a5f57 FAIL (2026-01-09 07:24:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001033838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 24.089884764s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:24:42 UTC) orbiter warning: context deadline exceeded Duration: 24.089907609s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.X.............X.....X....X...............................] duration: 5.754255847s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:24:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.754284923s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 5.000964883s trace: https://jaeger.dev.storj.io/trace/ecbaa70cad51c98 FAIL (2026-01-09 07:24:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000988338s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 18.5818345s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:23:59 UTC) orbiter warning: context deadline exceeded Duration: 18.581861683s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 5.00153919s trace: https://jaeger.dev.storj.io/trace/303a8f7635cf1d0 FAIL (2026-01-09 07:23:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00156032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X.............X.....X....X..............................] duration: 5.827101703s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:23:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.827127473s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 15.685657192s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:23:09 UTC) orbiter warning: context deadline exceeded Duration: 15.685684122s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration: 5.001332044s trace: https://jaeger.dev.storj.io/trace/47b00680403ca663 FAIL (2026-01-09 07:22:36 UTC) orbiter warning: context deadline exceeded Duration: 5.001354527s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 18.508970008s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:22:24 UTC) orbiter warning: context deadline exceeded Duration: 18.508987443s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.X.............X.....X....X.............................] duration: 5.869053894s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:22:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.869084523s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 16.339213397s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:21:26 UTC) orbiter warning: context deadline exceeded Duration: 16.339242573s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration: 5.001761253s trace: https://jaeger.dev.storj.io/trace/2a43aecd9619ac17 FAIL (2026-01-09 07:21:25 UTC) orbiter warning: context deadline exceeded Duration: 5.001784918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration: 5.001058534s trace: https://jaeger.dev.storj.io/trace/4351c12bd9cea39d FAIL (2026-01-09 07:21:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001151941s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X...X.X.............X.....X....X............................] duration: 5.6635977s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:20:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.663626244s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 7.500074981s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:20:47 UTC) orbiter warning: context deadline exceeded Duration: 7.500091602s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration: 5.001293422s trace: https://jaeger.dev.storj.io/trace/1b190f7280b7378f FAIL (2026-01-09 07:20:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001312038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 10.977526645s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:20:08 UTC) orbiter warning: context deadline exceeded Duration: 10.977589034s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 5.001209s trace: https://jaeger.dev.storj.io/trace/428d4a24f481a768 FAIL (2026-01-09 07:19:54 UTC) orbiter warning: context deadline exceeded Duration: 5.001229069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX...X.X.............X.....X....X...........................] duration: 5.99650579s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:19:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.996531799s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 17.312957218s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:19:22 UTC) orbiter warning: context deadline exceeded Duration: 17.312984129s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXX] duration: 5.001446774s trace: https://jaeger.dev.storj.io/trace/5d7cb853f41060ed FAIL (2026-01-09 07:18:46 UTC) orbiter warning: context deadline exceeded Duration: 5.001467544s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X......................X....................................] duration: 16.64985969s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:18:29 UTC) orbiter warning: context deadline exceeded Duration: 16.649882183s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XX...X.X.............X.....X....X..........................] duration: 5.747920905s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:18:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.747952395s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 5.001450791s trace: https://jaeger.dev.storj.io/trace/74753a053adde0e2 FAIL (2026-01-09 07:18:04 UTC) orbiter warning: context deadline exceeded Duration: 5.001475356s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X......................X...................................] duration: 16.899961478s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:17:44 UTC) orbiter warning: context deadline exceeded Duration: 16.899989081s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..XX...X.X.............X.....X....X.........................] duration: 5.818692745s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:17:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.81872202s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 5.000805293s trace: https://jaeger.dev.storj.io/trace/3cbb65db98de9c06 FAIL (2026-01-09 07:17:19 UTC) orbiter warning: context deadline exceeded Duration: 5.000820482s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X......................X..................................] duration: 16.029239802s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:17:09 UTC) orbiter warning: context deadline exceeded Duration: 16.029355824s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...XX...X.X.............X.....X....X........................] duration: 5.697042994s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:16:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.697071047s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 5.001145764s trace: https://jaeger.dev.storj.io/trace/7df0e74bc6ddb8fa FAIL (2026-01-09 07:16:36 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001164792s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..X......................X.................................] duration: 19.264752433s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:16:25 UTC) orbiter warning: context deadline exceeded Duration: 19.264777029s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....XX...X.X.............X.....X....X.......................] duration: 5.676505867s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:16:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.676541274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....XX...X.X.............X.....X....X......................] duration: 5.739514178s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:16:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.739544486s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration: 5.000647462s trace: https://jaeger.dev.storj.io/trace/301bd03c402b116 FAIL (2026-01-09 07:15:47 UTC) orbiter warning: context deadline exceeded Duration: 5.000668071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..X......................X................................] duration: 17.380207949s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:15:42 UTC) orbiter warning: context deadline exceeded Duration: 17.380234411s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......XX...X.X.............X.....X....X.....................] duration: 5.716299081s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:15:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.716327685s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......XX...X.X.............X.....X....X....................] duration: 5.825303719s trace: https://jaeger.dev.storj.io/trace/388fd50fe72a145e FAIL (2026-01-09 07:15:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== n 9 06:58:13.695272301 (831h29m13.566355346s) [upload=727286]: Starting upload Jan 9 06:58:13.695338286 (831h29m13.566421331s) [upload=727286]: Done starting upload Jan 9 06:58:13.695500856 (831h29m13.566583901s) [upload=727286]: Uploading segments... Jan 9 06:58:13.695769047 (831h29m13.566852082s) [upload=727286]: Got next segment. Inline: false Jan 9 06:58:13.695806127 (831h29m13.566889172s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 9 06:58:13.695812108 (831h29m13.566895144s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 9 06:58:13.945470344 (831h29m13.816553399s) [seg_pos={0 0}, upload=727286]: Begin upload segment... Jan 9 06:58:13.945521671 (831h29m13.816604706s) [seg_pos={0 0}, upload=727286]: Done begin upload segment. Jan 9 06:58:13.945559283 (831h29m13.816642328s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.945586395 (831h29m13.81666944s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946241853 (831h29m13.817324888s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.946662263 (831h29m13.817745308s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947149431 (831h29m13.818232486s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947586003 (831h29m13.818669048s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.947992246 (831h29m13.819075281s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948221212 (831h29m13.819304257s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948461410 (831h29m13.819544455s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948719861 (831h29m13.819802896s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.948898321 (831h29m13.819981356s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949200116 (831h29m13.820283161s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949314053 (831h29m13.820397098s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.949642358 (831h29m13.820725393s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950153292 (831h29m13.821236327s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:13.950643355 (831h29m13.8217264s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Uploading piece... Jan 9 06:58:15.655797138 (831h29m15.526880173s) [upload=727286]: Next returned nil segment Jan 9 06:58:15.655808910 (831h29m15.526891955s) [upload=727286]: Waiting for error group managing segments... Jan 9 06:58:15.936492946 (831h29m15.807575991s) [noise=true , address=145.79.227.3:4149 , piece_id=G6QOG3PIUO3TEZ7MUY33CKCVAQKNXQD2FJJXBFQS23U6VFFO2WNQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938126647 (831h29m15.809209692s) [noise=true , address=145.79.227.52:4446 , piece_id=6NGP26FGJWFVSQUKHFGMNYXK36Q2GKE3QMRMUAHMGBDIJTKRM42Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.938190048 (831h29m15.809273083s) [noise=true , address=145.79.227.37:4567 , piece_id=V5IFJPYOYADFTLQM4U462SDIR5YONXCRX5C5TECZRIEPQ5AGE7AQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941535648 (831h29m15.812618693s) [noise=true , address=145.79.227.11:4729 , piece_id=KITZ2ZCDYWYHL3V5XVTXX6LTD2JNYVSUXSSQBO7DJWCXNY6TQJ5Q, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.941599800 (831h29m15.812682835s) [noise=true , address=145.79.227.15:4876 , piece_id=IWQFL455354TREJSBNHBAE2N7D6FX34NXTFLLZ7ZTHTVLT4Q4O2A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950577376 (831h29m15.821660411s) [noise=true , address=145.79.227.7:4294 , piece_id=H72VIEACSLIG7BDQHJ3PZFEBLHFLC25MKCZNH5T3R2HISVTWWVOQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.950819457 (831h29m15.821902492s) [noise=true , address=145.79.227.50:4374 , piece_id=JDU2FNTABG2VIJIIRTZ5LDQVRXWUI44LXEP7IHTBMHMGSEQXCL6A, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.958061838 (831h29m15.829144883s) [noise=true , address=145.79.227.14:4837 , piece_id=XDLL3OXLGLQOW4ZRSSOIINCSSLN7ABAOGRAJF3ZA42NEKPKCKRWA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:15.959409655 (831h29m15.83049269s) [noise=true , address=145.79.227.35:4481 , piece_id=7GUTZCM7CONEZFZQ54OH4BD54LQSO6CYQEM3YRBAJYGRY2JCNUPQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.044492585 (831h29m15.91557562s) [noise=true , address=145.79.227.38:4977 , piece_id=BWDZ4FPKUXKMSU5YESRQ6CDJNYRBELY37Z2Y3YOR6EYZ3KGCGTZA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.238220553 (831h29m16.109303599s) [noise=true , address=77.90.31.13:4629 , piece_id=RFCRJEEIIDGDMQX3F26EYRKVYUZCAMB4KGEPKP6WAX2WYT55OUQQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.245229681 (831h29m16.116312726s) [noise=true , address=77.90.31.15:4697 , piece_id=LGDDNKCQLIHDQ6NKPGH2PDKVOVDE3VJXQUSZMKSKDEXQ4AMT72YQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357051368 (831h29m16.228134413s) [noise=true , address=77.90.31.14:4661 , piece_id=6BVCVUWUXP6CY4O6K5FTDEG4P6QZSJYKKU2DOJ7KTYQEPF2ZROZQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: <nil> Jan 9 06:58:16.357077027 (831h29m16.228160062s) [seg_pos={0 0}, upload=727286]: Segment reached optimal threshold of 13 pieces. Jan 9 06:58:16.357309239 (831h29m16.228392284s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357318416 (831h29m16.228401462s) [noise=true , address=45.91.116.5:4297 , piece_id=AB42Q7YQNKXP72WA7X47Z4FJ4BQF4FWCTBUZX264XFW6XMYFDEMQ, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357581948 (831h29m16.228664993s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Jan 9 06:58:16.357607006 (831h29m16.228690061s) [noise=true , address=45.91.116.10:4083 , piece_id=E3Y3AJDRWXKAMGFS2PT32BQ564RYIUNEFNP636VUFA5NIH5LERYA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357638056 (831h29m16.228721111s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46782->45.91.116.8:4047: use of closed network connection Jan 9 06:58:16.357650178 (831h29m16.228733233s) [noise=true , address=45.91.116.8:4047 , piece_id=7TZK3LF4VJBIEEHL47WWQAYP562DLROMGHH3QTXLVU2EW7XB6TDA, seg_pos={0 0}, upload=727286]: Long tail cancellation due to optimal threshold Jan 9 06:58:16.357673874 (831h29m16.228756909s) [seg_pos={0 0}, upload=727286]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 9 06:58:16.357759596 (831h29m16.228842652s) [upload=727286]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 9 06:58:16.357766750 (831h29m16.228849805s) [upload=727286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 9 06:58:16.644899615 (831h29m16.51598265s) [upload=727286]: Done uploading segments... Jan 9 06:58:16.644920265 (831h29m16.51600331s) [upload=727286]: Upload finished. err: <nil> Duration: 5.825339126s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.X..X......................X...............................] duration: 14.49537915s trace: https://jaeger.dev.storj.io/trace/4d4a7f7638a68428 FAIL (2026-01-09 07:14:52 UTC) orbiter warning: context deadline exceeded Duration: 14.495404909s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)