Orbiter History

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 472.074004ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:35:20 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 472.103019ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.465428889s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:35:10 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.465456451s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..XX.X...X.XX] duration: 30.003994885s trace: https://jaeger.dev.storj.io/trace/518b9dff40b34fc3 FAIL (2025-12-22 16:35:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: VZZ77FJ6OHR6CDASMVUUHFXSIFAFGJ43F7NCTGJC2FASF6NOX2LQ) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TI4CA4SEJSYRE23NHLNGP3GTH7MC3SIQA7BKSC7G5IETY5UWSJEQ) context deadline exceeded ======LOGS====== Dec 22 16:34:38.865105546 (409h5m38.736188591s) [upload=359138]: Starting upload Dec 22 16:34:38.865198393 (409h5m38.736281438s) [upload=359138]: Done starting upload Dec 22 16:34:38.865261052 (409h5m38.736344097s) [upload=359138]: Uploading segments... Dec 22 16:34:38.865376662 (409h5m38.736459707s) [upload=359138]: Got next segment. Inline: false Dec 22 16:34:38.865413843 (409h5m38.736496878s) [upload=359138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:34:38.865418572 (409h5m38.736501618s) [upload=359138]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:34:39.118933935 (409h5m38.99001698s) [seg_pos={0 0}, upload=359138]: Begin upload segment... Dec 22 16:34:39.118990002 (409h5m38.990073097s) [seg_pos={0 0}, upload=359138]: Done begin upload segment. Dec 22 16:34:39.119024628 (409h5m38.990107683s) [noise=true , address=145.79.227.45:4229 , piece_id=SDMZSS3UB7ZNDF5A6IROUM433RQ6OMD75F53ZYNCMTAXFHT2W6LQ, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.119087006 (409h5m38.990170051s) [noise=true , address=145.79.227.11:4729 , piece_id=E32SR7Z4N4VD33BGG5J5XKMB3SYNUTAIHPBH23VD6DWVPC3YNHWQ, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.119748065 (409h5m38.9908311s) [noise=true , address=45.91.116.14:4192 , piece_id=VZZ77FJ6OHR6CDASMVUUHFXSIFAFGJ43F7NCTGJC2FASF6NOX2LQ, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.120337377 (409h5m38.991420422s) [noise=true , address=145.79.227.51:4410 , piece_id=3SXZAF66QDCKLFLC5LNQQPMFCDVDCHJ3PEUNXCCMHI7LBLT4GECQ, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.120840305 (409h5m38.99192335s) [noise=true , address=45.91.116.11:4118 , piece_id=TI4CA4SEJSYRE23NHLNGP3GTH7MC3SIQA7BKSC7G5IETY5UWSJEQ, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.121218555 (409h5m38.992301601s) [noise=true , address=145.79.227.29:4943 , piece_id=NYDVBCNA4JTBBTLOVG7LVECIOTMZ3HVRYEUBRGSX4FFT367CN7UA, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.121604801 (409h5m38.992687846s) [noise=true , address=145.79.227.19:4078 , piece_id=JGE5JEZ336M3L54BJBX4VAJ2BLCGDHGMKRZIZ7G7LFCHRDOJR3VA, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.122062462 (409h5m38.993145497s) [noise=true , address=145.79.227.42:4123 , piece_id=JBU3BSE3HEE2H4QBLI6SPVDK6ZPOXIXQKD3OZWDUN67D72E3YCFA, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.122275638 (409h5m38.993358673s) [noise=true , address=45.91.116.10:4084 , piece_id=LDBNC7DNI4VNGC2VWJPQHKMPL2CGMMQ7XZ52VTP2Z3CJVTUBICYA, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.122494244 (409h5m38.993577289s) [noise=true , address=45.91.116.12:4227 , piece_id=AZG7KPHIRJBXT57H5AVITJFZDMDV2NG5W2B4L3YMFBNTTFRR74QA, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.122842988 (409h5m38.993926023s) [noise=true , address=145.79.227.41:4085 , piece_id=Q67B7VIKVQPD77U3LVRXHLWP3UCMO2BLOENNZQYB3OMD3BEOGNNQ, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.123546938 (409h5m38.994629993s) [noise=true , address=77.90.31.14:4663 , piece_id=BJKCBW7NG4WEHJNFXOI6GZB5M2SQI2JU7J63VIFP2ADUQPWXGJAQ, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.124061859 (409h5m38.995144904s) [noise=true , address=145.79.227.52:4446 , piece_id=7PRRWVWTT6SMCMKCMWIWZN3E2WUH4BLBB33UXMFGUKHH3HBMVJWQ, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.124591267 (409h5m38.995674302s) [noise=true , address=145.79.227.33:4870 , piece_id=7WEYDVBJP7ZP6GH52WLE4IVOZQNFDHQL6YDLEPWBFY2NIKV6XGGQ, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.124907640 (409h5m38.995990685s) [noise=true , address=145.79.227.39:4015 , piece_id=UOPWIGUP2J62LLV4R6DD3TN34VWUUUC2SQZVRU67ELLGO52D25MA, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:39.125379488 (409h5m38.996462533s) [noise=true , address=145.79.227.37:4567 , piece_id=AKRROU35QK36WIAMPIND5ETGJFGOKF2PNAEW3E4ATGMPMDA7HAIA, seg_pos={0 0}, upload=359138]: Uploading piece... Dec 22 16:34:41.056669971 (409h5m40.927753006s) [upload=359138]: Next returned nil segment Dec 22 16:34:41.056681954 (409h5m40.927764989s) [upload=359138]: Waiting for error group managing segments... Dec 22 16:34:41.361416146 (409h5m41.232499191s) [noise=true , address=77.90.31.14:4663 , piece_id=BJKCBW7NG4WEHJNFXOI6GZB5M2SQI2JU7J63VIFP2ADUQPWXGJAQ, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.383826800 (409h5m41.254909845s) [noise=true , address=45.91.116.10:4084 , piece_id=LDBNC7DNI4VNGC2VWJPQHKMPL2CGMMQ7XZ52VTP2Z3CJVTUBICYA, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.383939644 (409h5m41.25502268s) [noise=true , address=45.91.116.11:4118 , piece_id=TI4CA4SEJSYRE23NHLNGP3GTH7MC3SIQA7BKSC7G5IETY5UWSJEQ, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.405741839 (409h5m41.276824874s) [noise=true , address=45.91.116.14:4192 , piece_id=VZZ77FJ6OHR6CDASMVUUHFXSIFAFGJ43F7NCTGJC2FASF6NOX2LQ, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.448073396 (409h5m41.319156431s) [noise=true , address=45.91.116.12:4227 , piece_id=AZG7KPHIRJBXT57H5AVITJFZDMDV2NG5W2B4L3YMFBNTTFRR74QA, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.911168492 (409h5m41.782251527s) [noise=true , address=145.79.227.51:4410 , piece_id=3SXZAF66QDCKLFLC5LNQQPMFCDVDCHJ3PEUNXCCMHI7LBLT4GECQ, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.933309120 (409h5m41.804392155s) [noise=true , address=145.79.227.39:4015 , piece_id=UOPWIGUP2J62LLV4R6DD3TN34VWUUUC2SQZVRU67ELLGO52D25MA, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.942007794 (409h5m41.813090829s) [noise=true , address=145.79.227.29:4943 , piece_id=NYDVBCNA4JTBBTLOVG7LVECIOTMZ3HVRYEUBRGSX4FFT367CN7UA, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.944169931 (409h5m41.815252976s) [noise=true , address=145.79.227.11:4729 , piece_id=E32SR7Z4N4VD33BGG5J5XKMB3SYNUTAIHPBH23VD6DWVPC3YNHWQ, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.967984859 (409h5m41.839067904s) [noise=true , address=145.79.227.41:4085 , piece_id=Q67B7VIKVQPD77U3LVRXHLWP3UCMO2BLOENNZQYB3OMD3BEOGNNQ, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.972759260 (409h5m41.843842305s) [noise=true , address=145.79.227.19:4078 , piece_id=JGE5JEZ336M3L54BJBX4VAJ2BLCGDHGMKRZIZ7G7LFCHRDOJR3VA, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.977950034 (409h5m41.849033069s) [noise=true , address=145.79.227.52:4446 , piece_id=7PRRWVWTT6SMCMKCMWIWZN3E2WUH4BLBB33UXMFGUKHH3HBMVJWQ, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.978737263 (409h5m41.849820298s) [noise=true , address=145.79.227.37:4567 , piece_id=AKRROU35QK36WIAMPIND5ETGJFGOKF2PNAEW3E4ATGMPMDA7HAIA, seg_pos={0 0}, upload=359138]: Done uploading piece. err: <nil> Dec 22 16:34:41.978756841 (409h5m41.849839876s) [seg_pos={0 0}, upload=359138]: Segment reached optimal threshold of 13 pieces. Dec 22 16:34:41.978977431 (409h5m41.850060466s) [noise=true , address=145.79.227.45:4229 , piece_id=SDMZSS3UB7ZNDF5A6IROUM433RQ6OMD75F53ZYNCMTAXFHT2W6LQ, seg_pos={0 0}, upload=359138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wWYnhWHSqqAFgje4p9Cr7WYaJ6dJWsCqvrx8ZqVFoakjuFeka): protocol: expected piece hash; context canceled; EOF Dec 22 16:34:41.978984444 (409h5m41.850067489s) [noise=true , address=145.79.227.45:4229 , piece_id=SDMZSS3UB7ZNDF5A6IROUM433RQ6OMD75F53ZYNCMTAXFHT2W6LQ, seg_pos={0 0}, upload=359138]: Long tail cancellation due to optimal threshold Dec 22 16:34:41.979185988 (409h5m41.850269033s) [noise=true , address=145.79.227.42:4123 , piece_id=JBU3BSE3HEE2H4QBLI6SPVDK6ZPOXIXQKD3OZWDUN67D72E3YCFA, seg_pos={0 0}, upload=359138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM): protocol: expected piece hash; context canceled; EOF Dec 22 16:34:41.979193472 (409h5m41.850276507s) [noise=true , address=145.79.227.33:4870 , piece_id=7WEYDVBJP7ZP6GH52WLE4IVOZQNFDHQL6YDLEPWBFY2NIKV6XGGQ, seg_pos={0 0}, upload=359138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 22 16:34:41.979208150 (409h5m41.850291185s) [noise=true , address=145.79.227.33:4870 , piece_id=7WEYDVBJP7ZP6GH52WLE4IVOZQNFDHQL6YDLEPWBFY2NIKV6XGGQ, seg_pos={0 0}, upload=359138]: Long tail cancellation due to optimal threshold Dec 22 16:34:41.979224040 (409h5m41.850307085s) [noise=true , address=145.79.227.42:4123 , piece_id=JBU3BSE3HEE2H4QBLI6SPVDK6ZPOXIXQKD3OZWDUN67D72E3YCFA, seg_pos={0 0}, upload=359138]: Long tail cancellation due to optimal threshold Dec 22 16:34:41.979275948 (409h5m41.850358994s) [seg_pos={0 0}, upload=359138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:34:41.979369287 (409h5m41.850452322s) [upload=359138]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:34:41.979374456 (409h5m41.850457491s) [upload=359138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:34:42.288517012 (409h5m42.159600067s) [upload=359138]: Done uploading segments... Dec 22 16:34:42.288541129 (409h5m42.159624174s) [upload=359138]: Upload finished. err: <nil> Duration: 30.004022127s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.091389725s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:35:04 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.091423157s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 480.98975ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:35:03 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 481.020449ms core satellite eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 466.548292ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:34:41 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 466.57393ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.447277276s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:34:17 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.447297504s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..XX.X...X.X] duration: 30.002273502s trace: https://jaeger.dev.storj.io/trace/6dba7646f4695c53 FAIL (2025-12-22 16:34:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 16:33:38.490158348 (409h4m38.361241383s) [upload=359125]: Starting upload Dec 22 16:33:38.490251004 (409h4m38.361334039s) [upload=359125]: Done starting upload Dec 22 16:33:38.490385661 (409h4m38.361468706s) [upload=359125]: Uploading segments... Dec 22 16:33:38.490477906 (409h4m38.361560941s) [upload=359125]: Got next segment. Inline: false Dec 22 16:33:38.490505989 (409h4m38.361589035s) [upload=359125]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:33:38.490519585 (409h4m38.36160262s) [upload=359125]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:33:38.848328270 (409h4m38.719411305s) [seg_pos={0 0}, upload=359125]: Begin upload segment... Dec 22 16:33:38.848373035 (409h4m38.71945608s) [seg_pos={0 0}, upload=359125]: Done begin upload segment. Dec 22 16:33:38.848398423 (409h4m38.719481458s) [noise=true , address=45.91.116.4:4940 , piece_id=FO3LQGUHCZELZMOOBC4LTZE4Z6TGMXMJ6SUEBWCR73LHQJFUJH5Q, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.848407300 (409h4m38.719490345s) [noise=true , address=145.79.227.49:4339 , piece_id=L5W73DQTHSIDGUMPUVPSCFETFHWVRCAFJCKT62A4NPFHC252QCCA, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.848525937 (409h4m38.719608982s) [noise=true , address=145.79.227.7:4294 , piece_id=6JGZC24BAHZYA3WOVJO4D3NFX43PEDSFQJHHNCNZEEKA4HHA6MVA, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.848582884 (409h4m38.719665919s) [noise=true , address=145.79.227.50:4375 , piece_id=A5I6NV7R6M6KIYORXHPUE2GOHX4VKGD5EW6Z6ILEVBVQHFXTKI2Q, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.848953631 (409h4m38.720036666s) [noise=true , address=77.90.31.11:4553 , piece_id=U5WDPDFXAWXOHEAYYPAFF5FSOWLSTZOR4MBOHVXBOL3RZGQUUKFQ, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.849116451 (409h4m38.720199486s) [noise=true , address=145.79.227.11:4732 , piece_id=LO2PXVHUX6F5AUMM4WC2PNYDC34YTY3GHRLCRLV6BQWAVZABW36Q, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.849462710 (409h4m38.720545755s) [noise=true , address=145.79.227.15:4875 , piece_id=UX4LD7VKM6E763XFWRYUTFUZ6CUH64BMTVJVFMCUNRCAIIFOBRZA, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.849728416 (409h4m38.720811451s) [noise=true , address=145.79.227.51:4410 , piece_id=RF2QV5KKY4P6ZZCNISVWE2FGYMYMMSJLHTZXLOLI6U33R6CHTVAQ, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.850149177 (409h4m38.721232222s) [noise=true , address=45.91.116.8:4045 , piece_id=NDBGFQNNJPDQRAMKOW33JVLSWQVR56Q6NM4XN7JAQEVEJRVGZWCA, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.850635844 (409h4m38.721718879s) [noise=true , address=45.91.116.10:4084 , piece_id=WOTAP53LHNRBPVFRKVALAV5INY656F5MCNSOA3NCYWTBL2XNJGQA, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.850975721 (409h4m38.722058766s) [noise=true , address=145.79.227.39:4014 , piece_id=LYABV6S3NTLJI5WYEGFTCFDG5H4W25HN5C6XM7DBAJFDUSDWRT2Q, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.851405539 (409h4m38.722488584s) [noise=true , address=145.79.227.42:4122 , piece_id=GNPHGXVXO7XAGFC6WAIZTXD7KU6BZ6TBZE5HUBMWBUGUN7LH2XCA, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.851933615 (409h4m38.723016651s) [noise=true , address=45.91.116.13:4156 , piece_id=MU2PPBRNG2GIEPSIOKE7WS32WGAXQY3ZMWDBNJEIQUX35PDR6CDQ, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.852326423 (409h4m38.723409468s) [noise=true , address=145.79.227.9:4366 , piece_id=TCXMP5HXVRNOABOUGQU7B3EKNJKDZ625EYVI7JCNYITSJE5BBDFQ, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.852660709 (409h4m38.723743744s) [noise=true , address=145.79.227.33:4870 , piece_id=KO6KX2EQXNUHW6IHSMFPKAUNQYVDY7OT2CQI3R6BCZDT2FMDMKKA, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:38.852967704 (409h4m38.724050739s) [noise=true , address=45.91.116.5:4300 , piece_id=MEY5HBUHAFPDGUHOKNZA4OE2LGGTFQBG6352Z54QQMG7X3OEEFSQ, seg_pos={0 0}, upload=359125]: Uploading piece... Dec 22 16:33:40.729913536 (409h4m40.600996581s) [upload=359125]: Next returned nil segment Dec 22 16:33:40.729970545 (409h4m40.60105358s) [upload=359125]: Waiting for error group managing segments... Dec 22 16:33:41.109588849 (409h4m40.980671884s) [noise=true , address=45.91.116.8:4045 , piece_id=NDBGFQNNJPDQRAMKOW33JVLSWQVR56Q6NM4XN7JAQEVEJRVGZWCA, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.112110400 (409h4m40.983193435s) [noise=true , address=45.91.116.10:4084 , piece_id=WOTAP53LHNRBPVFRKVALAV5INY656F5MCNSOA3NCYWTBL2XNJGQA, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.127062661 (409h4m40.998145706s) [noise=true , address=45.91.116.4:4940 , piece_id=FO3LQGUHCZELZMOOBC4LTZE4Z6TGMXMJ6SUEBWCR73LHQJFUJH5Q, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.182516173 (409h4m41.053599208s) [noise=true , address=45.91.116.13:4156 , piece_id=MU2PPBRNG2GIEPSIOKE7WS32WGAXQY3ZMWDBNJEIQUX35PDR6CDQ, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.188728483 (409h4m41.059811528s) [noise=true , address=45.91.116.5:4300 , piece_id=MEY5HBUHAFPDGUHOKNZA4OE2LGGTFQBG6352Z54QQMG7X3OEEFSQ, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.736532448 (409h4m41.607615494s) [noise=true , address=145.79.227.11:4732 , piece_id=LO2PXVHUX6F5AUMM4WC2PNYDC34YTY3GHRLCRLV6BQWAVZABW36Q, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.738813832 (409h4m41.609896877s) [noise=true , address=145.79.227.9:4366 , piece_id=TCXMP5HXVRNOABOUGQU7B3EKNJKDZ625EYVI7JCNYITSJE5BBDFQ, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.740990567 (409h4m41.612073603s) [noise=true , address=145.79.227.15:4875 , piece_id=UX4LD7VKM6E763XFWRYUTFUZ6CUH64BMTVJVFMCUNRCAIIFOBRZA, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.746324955 (409h4m41.61740799s) [noise=true , address=145.79.227.50:4375 , piece_id=A5I6NV7R6M6KIYORXHPUE2GOHX4VKGD5EW6Z6ILEVBVQHFXTKI2Q, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.782428983 (409h4m41.653512018s) [noise=true , address=145.79.227.39:4014 , piece_id=LYABV6S3NTLJI5WYEGFTCFDG5H4W25HN5C6XM7DBAJFDUSDWRT2Q, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.795328455 (409h4m41.66641149s) [noise=true , address=145.79.227.51:4410 , piece_id=RF2QV5KKY4P6ZZCNISVWE2FGYMYMMSJLHTZXLOLI6U33R6CHTVAQ, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.803228779 (409h4m41.674311824s) [noise=true , address=145.79.227.42:4122 , piece_id=GNPHGXVXO7XAGFC6WAIZTXD7KU6BZ6TBZE5HUBMWBUGUN7LH2XCA, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.808195756 (409h4m41.679278801s) [noise=true , address=77.90.31.11:4553 , piece_id=U5WDPDFXAWXOHEAYYPAFF5FSOWLSTZOR4MBOHVXBOL3RZGQUUKFQ, seg_pos={0 0}, upload=359125]: Done uploading piece. err: <nil> Dec 22 16:33:41.808218850 (409h4m41.679301886s) [seg_pos={0 0}, upload=359125]: Segment reached optimal threshold of 13 pieces. Dec 22 16:33:41.808464839 (409h4m41.679547874s) [noise=true , address=145.79.227.49:4339 , piece_id=L5W73DQTHSIDGUMPUVPSCFETFHWVRCAFJCKT62A4NPFHC252QCCA, seg_pos={0 0}, upload=359125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:55532->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:55532->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55532->145.79.227.49:4339: use of closed network connection Dec 22 16:33:41.808473435 (409h4m41.67955648s) [noise=true , address=145.79.227.49:4339 , piece_id=L5W73DQTHSIDGUMPUVPSCFETFHWVRCAFJCKT62A4NPFHC252QCCA, seg_pos={0 0}, upload=359125]: Long tail cancellation due to optimal threshold Dec 22 16:33:41.808601038 (409h4m41.679684093s) [noise=true , address=145.79.227.7:4294 , piece_id=6JGZC24BAHZYA3WOVJO4D3NFX43PEDSFQJHHNCNZEEKA4HHA6MVA, seg_pos={0 0}, upload=359125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: expected piece hash; context canceled; EOF Dec 22 16:33:41.808613441 (409h4m41.679696487s) [noise=true , address=145.79.227.7:4294 , piece_id=6JGZC24BAHZYA3WOVJO4D3NFX43PEDSFQJHHNCNZEEKA4HHA6MVA, seg_pos={0 0}, upload=359125]: Long tail cancellation due to optimal threshold Dec 22 16:33:41.808817610 (409h4m41.679900676s) [noise=true , address=145.79.227.33:4870 , piece_id=KO6KX2EQXNUHW6IHSMFPKAUNQYVDY7OT2CQI3R6BCZDT2FMDMKKA, seg_pos={0 0}, upload=359125]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 22 16:33:41.808827218 (409h4m41.679910263s) [noise=true , address=145.79.227.33:4870 , piece_id=KO6KX2EQXNUHW6IHSMFPKAUNQYVDY7OT2CQI3R6BCZDT2FMDMKKA, seg_pos={0 0}, upload=359125]: Long tail cancellation due to optimal threshold Dec 22 16:33:41.808850433 (409h4m41.679933478s) [seg_pos={0 0}, upload=359125]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:33:41.808938311 (409h4m41.680021356s) [upload=359125]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:33:41.808944492 (409h4m41.680027537s) [upload=359125]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:33:42.201462399 (409h4m42.072545435s) [upload=359125]: Done uploading segments... Dec 22 16:33:42.201484992 (409h4m42.072568027s) [upload=359125]: Upload finished. err: <nil> Duration: 30.002297206s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.930845145s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:34:07 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.930877136s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.387493657s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:33:55 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.38752692s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.18794507s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:33:48 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.187964116s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 477.735764ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:33:37 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 477.758298ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.503576066s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:33:22 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.503599732s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 466.617143ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:33:21 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 466.644685ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..XX.X...X.] duration: 10.699248443s trace: https://jaeger.dev.storj.io/trace/3ea32d400380c3ca FAIL (2025-12-22 16:33:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 16:31:54.463884215 (409h2m54.33496726s) [upload=359102]: Starting upload Dec 22 16:31:54.463971440 (409h2m54.335054485s) [upload=359102]: Done starting upload Dec 22 16:31:54.464151835 (409h2m54.33523488s) [upload=359102]: Uploading segments... Dec 22 16:31:54.464187022 (409h2m54.335270057s) [upload=359102]: Got next segment. Inline: false Dec 22 16:31:54.464210987 (409h2m54.335294022s) [upload=359102]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:31:54.464214904 (409h2m54.335297939s) [upload=359102]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:31:54.789988479 (409h2m54.661071544s) [seg_pos={0 0}, upload=359102]: Begin upload segment... Dec 22 16:31:54.790124618 (409h2m54.661207663s) [seg_pos={0 0}, upload=359102]: Done begin upload segment. Dec 22 16:31:54.790171467 (409h2m54.661254502s) [noise=true , address=145.79.227.39:4015 , piece_id=YUWO5NRBGZAYMOAULJECX7EBTKAH5UZU3IQWW7JJUTJFLHRIXD7Q, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.790180955 (409h2m54.661264s) [noise=true , address=145.79.227.43:4157 , piece_id=IAMJUVMZXDJB23XYKC3W777NVQ7AK7V66TRYTU5T6FFH53DI6TOA, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.790250297 (409h2m54.661333342s) [noise=true , address=145.79.227.16:4911 , piece_id=TBC4S2PNDODYPZLRWSMI3EOVBLPSP2W5CKI7SHLCSV2ENXSRCEZQ, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.790754066 (409h2m54.661837112s) [noise=true , address=145.79.227.50:4374 , piece_id=KO26353DG3WVV6KCJK4MSMEPMWQHMDJ7R4UH4DWIDQMVO3WX2OQA, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.791305756 (409h2m54.662388791s) [noise=true , address=77.90.31.11:4556 , piece_id=DC3BGKA6Q5GL564MLRTDBRQI4RULCQ5TSB7GUQB2BEAUMBPBFJ5A, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.792039102 (409h2m54.663122138s) [noise=true , address=45.91.116.10:4081 , piece_id=XMFFDGEDCSYZEFMJALHCIAQCYRMHFXXI6PHKIKQYWE775FXOXPRQ, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.792554785 (409h2m54.66363783s) [noise=true , address=145.79.227.48:4301 , piece_id=PELYPBNNTMBONPHJEGXPBROIQCSQFTVDH5NGFZ2O6BFYPFWN2XKA, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.793159285 (409h2m54.66424234s) [noise=true , address=145.79.227.37:4566 , piece_id=VDFSVSQF5K3OEFHHROWBXLQVWCOPJDDJMVTXOJPGRIVCVEALHOUQ, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.794967018 (409h2m54.666050063s) [noise=true , address=45.91.116.11:4117 , piece_id=PDIFRYFWAIXQKDLSAAHXWN42OOSS2CW6UDCKVBHRDSCJ6REGCG7A, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.795111833 (409h2m54.666194878s) [noise=true , address=145.79.227.9:4366 , piece_id=SIWOUBIJIX2KK5ZZSS7M4LDQ3PBPUWUXV4MYPNUAP6DBCCUSH4UQ, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.796818222 (409h2m54.667901258s) [noise=true , address=145.79.227.20:4438 , piece_id=DW43VK3SXFFF2SHNEX62RP65EQNSVNEAFWGHZTYCFIAGZLHNXSWQ, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.798514783 (409h2m54.669597818s) [noise=true , address=77.90.31.17:4773 , piece_id=CIPGG7PKHE3DIMGOG3FLYZ5L4BNDUWT7GXIR7PZ4IMQXLE4S4HRA, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.800164896 (409h2m54.671247941s) [noise=true , address=45.91.116.6:4975 , piece_id=SEP6DNRO6UNERV5BAEUKYN4HI64XEGDRMSFZOPLZF5MFA5SCO52Q, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.801777717 (409h2m54.672860752s) [noise=true , address=145.79.227.5:4221 , piece_id=BRFZQIFCO5H23N7WRQCMALDQJYIU2RVTKZ5QPR4FMXYXNZ7AIY7A, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.804061104 (409h2m54.675144139s) [noise=true , address=145.79.227.33:4871 , piece_id=LETKGEFTDBTLOJ2AUG2R6YLM2EKXFNNBVONGR5GQ7ROAQ3MJFOFA, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.806804438 (409h2m54.677887473s) [noise=true , address=45.91.116.14:4190 , piece_id=C4KLRDIZUDGV42PR7IT6U566JWI4Y74GTQYIQWXDZD23OZXQJN3A, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:56.629918965 (409h2m56.501002s) [upload=359102]: Next returned nil segment Dec 22 16:31:56.629932942 (409h2m56.501015977s) [upload=359102]: Waiting for error group managing segments... Dec 22 16:31:56.983313755 (409h2m56.8543968s) [noise=true , address=77.90.31.11:4556 , piece_id=DC3BGKA6Q5GL564MLRTDBRQI4RULCQ5TSB7GUQB2BEAUMBPBFJ5A, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.006602439 (409h2m56.877685474s) [noise=true , address=45.91.116.11:4117 , piece_id=PDIFRYFWAIXQKDLSAAHXWN42OOSS2CW6UDCKVBHRDSCJ6REGCG7A, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.029221380 (409h2m56.900304425s) [noise=true , address=77.90.31.17:4773 , piece_id=CIPGG7PKHE3DIMGOG3FLYZ5L4BNDUWT7GXIR7PZ4IMQXLE4S4HRA, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.053639015 (409h2m56.92472206s) [noise=true , address=45.91.116.10:4081 , piece_id=XMFFDGEDCSYZEFMJALHCIAQCYRMHFXXI6PHKIKQYWE775FXOXPRQ, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.099590674 (409h2m56.970673709s) [noise=true , address=45.91.116.6:4975 , piece_id=SEP6DNRO6UNERV5BAEUKYN4HI64XEGDRMSFZOPLZF5MFA5SCO52Q, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.100101136 (409h2m56.971184171s) [noise=true , address=45.91.116.14:4190 , piece_id=C4KLRDIZUDGV42PR7IT6U566JWI4Y74GTQYIQWXDZD23OZXQJN3A, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.584867928 (409h2m57.455950963s) [noise=true , address=145.79.227.50:4374 , piece_id=KO26353DG3WVV6KCJK4MSMEPMWQHMDJ7R4UH4DWIDQMVO3WX2OQA, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.604114867 (409h2m57.475197902s) [noise=true , address=145.79.227.39:4015 , piece_id=YUWO5NRBGZAYMOAULJECX7EBTKAH5UZU3IQWW7JJUTJFLHRIXD7Q, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.608206619 (409h2m57.479289654s) [noise=true , address=145.79.227.43:4157 , piece_id=IAMJUVMZXDJB23XYKC3W777NVQ7AK7V66TRYTU5T6FFH53DI6TOA, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.608293174 (409h2m57.479376219s) [noise=true , address=145.79.227.9:4366 , piece_id=SIWOUBIJIX2KK5ZZSS7M4LDQ3PBPUWUXV4MYPNUAP6DBCCUSH4UQ, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.631338686 (409h2m57.502421721s) [noise=true , address=145.79.227.33:4871 , piece_id=LETKGEFTDBTLOJ2AUG2R6YLM2EKXFNNBVONGR5GQ7ROAQ3MJFOFA, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.675430132 (409h2m57.546513167s) [noise=true , address=145.79.227.16:4911 , piece_id=TBC4S2PNDODYPZLRWSMI3EOVBLPSP2W5CKI7SHLCSV2ENXSRCEZQ, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.883038140 (409h2m57.754121175s) [noise=true , address=145.79.227.48:4301 , piece_id=PELYPBNNTMBONPHJEGXPBROIQCSQFTVDH5NGFZ2O6BFYPFWN2XKA, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.883057828 (409h2m57.754140873s) [seg_pos={0 0}, upload=359102]: Segment reached optimal threshold of 13 pieces. Dec 22 16:31:57.883249703 (409h2m57.754332738s) [noise=true , address=145.79.227.37:4566 , piece_id=VDFSVSQF5K3OEFHHROWBXLQVWCOPJDDJMVTXOJPGRIVCVEALHOUQ, seg_pos={0 0}, upload=359102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 16:31:57.883255545 (409h2m57.7543386s) [noise=true , address=145.79.227.37:4566 , piece_id=VDFSVSQF5K3OEFHHROWBXLQVWCOPJDDJMVTXOJPGRIVCVEALHOUQ, seg_pos={0 0}, upload=359102]: Long tail cancellation due to optimal threshold Dec 22 16:31:57.883280411 (409h2m57.754363446s) [noise=true , address=145.79.227.20:4438 , piece_id=DW43VK3SXFFF2SHNEX62RP65EQNSVNEAFWGHZTYCFIAGZLHNXSWQ, seg_pos={0 0}, upload=359102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 22 16:31:57.883288908 (409h2m57.754371953s) [noise=true , address=145.79.227.20:4438 , piece_id=DW43VK3SXFFF2SHNEX62RP65EQNSVNEAFWGHZTYCFIAGZLHNXSWQ, seg_pos={0 0}, upload=359102]: Long tail cancellation due to optimal threshold Dec 22 16:31:57.883366205 (409h2m57.75444924s) [noise=true , address=145.79.227.5:4221 , piece_id=BRFZQIFCO5H23N7WRQCMALDQJYIU2RVTKZ5QPR4FMXYXNZ7AIY7A, seg_pos={0 0}, upload=359102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:57766->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:57766->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57766->145.79.227.5:4221: use of closed network connection Dec 22 16:31:57.883371866 (409h2m57.754454911s) [noise=true , address=145.79.227.5:4221 , piece_id=BRFZQIFCO5H23N7WRQCMALDQJYIU2RVTKZ5QPR4FMXYXNZ7AIY7A, seg_pos={0 0}, upload=359102]: Long tail cancellation due to optimal threshold Dec 22 16:31:57.883392114 (409h2m57.754475149s) [seg_pos={0 0}, upload=359102]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:31:57.883473209 (409h2m57.754556244s) [upload=359102]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:31:57.883480342 (409h2m57.754563377s) [upload=359102]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:31:58.277921430 (409h2m58.149004465s) [upload=359102]: Done uploading segments... Dec 22 16:31:58.277938593 (409h2m58.149021628s) [upload=359102]: Upload finished. err: <nil> Duration: 10.699273972s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 464.847744ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:33:06 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 464.879094ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.166839851s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:33:05 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.166892572s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.446843934s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:32:50 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.446867719s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.445811133s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:32:45 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.445839828s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 473.178125ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:32:32 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 473.209415ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..XX.X...X] duration: 30.004790103s trace: https://jaeger.dev.storj.io/trace/3ea32d400380c3ca FAIL (2025-12-22 16:32:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 16:31:54.463884215 (409h2m54.33496726s) [upload=359102]: Starting upload Dec 22 16:31:54.463971440 (409h2m54.335054485s) [upload=359102]: Done starting upload Dec 22 16:31:54.464151835 (409h2m54.33523488s) [upload=359102]: Uploading segments... Dec 22 16:31:54.464187022 (409h2m54.335270057s) [upload=359102]: Got next segment. Inline: false Dec 22 16:31:54.464210987 (409h2m54.335294022s) [upload=359102]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:31:54.464214904 (409h2m54.335297939s) [upload=359102]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:31:54.789988479 (409h2m54.661071544s) [seg_pos={0 0}, upload=359102]: Begin upload segment... Dec 22 16:31:54.790124618 (409h2m54.661207663s) [seg_pos={0 0}, upload=359102]: Done begin upload segment. Dec 22 16:31:54.790171467 (409h2m54.661254502s) [noise=true , address=145.79.227.39:4015 , piece_id=YUWO5NRBGZAYMOAULJECX7EBTKAH5UZU3IQWW7JJUTJFLHRIXD7Q, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.790180955 (409h2m54.661264s) [noise=true , address=145.79.227.43:4157 , piece_id=IAMJUVMZXDJB23XYKC3W777NVQ7AK7V66TRYTU5T6FFH53DI6TOA, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.790250297 (409h2m54.661333342s) [noise=true , address=145.79.227.16:4911 , piece_id=TBC4S2PNDODYPZLRWSMI3EOVBLPSP2W5CKI7SHLCSV2ENXSRCEZQ, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.790754066 (409h2m54.661837112s) [noise=true , address=145.79.227.50:4374 , piece_id=KO26353DG3WVV6KCJK4MSMEPMWQHMDJ7R4UH4DWIDQMVO3WX2OQA, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.791305756 (409h2m54.662388791s) [noise=true , address=77.90.31.11:4556 , piece_id=DC3BGKA6Q5GL564MLRTDBRQI4RULCQ5TSB7GUQB2BEAUMBPBFJ5A, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.792039102 (409h2m54.663122138s) [noise=true , address=45.91.116.10:4081 , piece_id=XMFFDGEDCSYZEFMJALHCIAQCYRMHFXXI6PHKIKQYWE775FXOXPRQ, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.792554785 (409h2m54.66363783s) [noise=true , address=145.79.227.48:4301 , piece_id=PELYPBNNTMBONPHJEGXPBROIQCSQFTVDH5NGFZ2O6BFYPFWN2XKA, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.793159285 (409h2m54.66424234s) [noise=true , address=145.79.227.37:4566 , piece_id=VDFSVSQF5K3OEFHHROWBXLQVWCOPJDDJMVTXOJPGRIVCVEALHOUQ, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.794967018 (409h2m54.666050063s) [noise=true , address=45.91.116.11:4117 , piece_id=PDIFRYFWAIXQKDLSAAHXWN42OOSS2CW6UDCKVBHRDSCJ6REGCG7A, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.795111833 (409h2m54.666194878s) [noise=true , address=145.79.227.9:4366 , piece_id=SIWOUBIJIX2KK5ZZSS7M4LDQ3PBPUWUXV4MYPNUAP6DBCCUSH4UQ, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.796818222 (409h2m54.667901258s) [noise=true , address=145.79.227.20:4438 , piece_id=DW43VK3SXFFF2SHNEX62RP65EQNSVNEAFWGHZTYCFIAGZLHNXSWQ, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.798514783 (409h2m54.669597818s) [noise=true , address=77.90.31.17:4773 , piece_id=CIPGG7PKHE3DIMGOG3FLYZ5L4BNDUWT7GXIR7PZ4IMQXLE4S4HRA, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.800164896 (409h2m54.671247941s) [noise=true , address=45.91.116.6:4975 , piece_id=SEP6DNRO6UNERV5BAEUKYN4HI64XEGDRMSFZOPLZF5MFA5SCO52Q, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.801777717 (409h2m54.672860752s) [noise=true , address=145.79.227.5:4221 , piece_id=BRFZQIFCO5H23N7WRQCMALDQJYIU2RVTKZ5QPR4FMXYXNZ7AIY7A, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.804061104 (409h2m54.675144139s) [noise=true , address=145.79.227.33:4871 , piece_id=LETKGEFTDBTLOJ2AUG2R6YLM2EKXFNNBVONGR5GQ7ROAQ3MJFOFA, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:54.806804438 (409h2m54.677887473s) [noise=true , address=45.91.116.14:4190 , piece_id=C4KLRDIZUDGV42PR7IT6U566JWI4Y74GTQYIQWXDZD23OZXQJN3A, seg_pos={0 0}, upload=359102]: Uploading piece... Dec 22 16:31:56.629918965 (409h2m56.501002s) [upload=359102]: Next returned nil segment Dec 22 16:31:56.629932942 (409h2m56.501015977s) [upload=359102]: Waiting for error group managing segments... Dec 22 16:31:56.983313755 (409h2m56.8543968s) [noise=true , address=77.90.31.11:4556 , piece_id=DC3BGKA6Q5GL564MLRTDBRQI4RULCQ5TSB7GUQB2BEAUMBPBFJ5A, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.006602439 (409h2m56.877685474s) [noise=true , address=45.91.116.11:4117 , piece_id=PDIFRYFWAIXQKDLSAAHXWN42OOSS2CW6UDCKVBHRDSCJ6REGCG7A, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.029221380 (409h2m56.900304425s) [noise=true , address=77.90.31.17:4773 , piece_id=CIPGG7PKHE3DIMGOG3FLYZ5L4BNDUWT7GXIR7PZ4IMQXLE4S4HRA, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.053639015 (409h2m56.92472206s) [noise=true , address=45.91.116.10:4081 , piece_id=XMFFDGEDCSYZEFMJALHCIAQCYRMHFXXI6PHKIKQYWE775FXOXPRQ, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.099590674 (409h2m56.970673709s) [noise=true , address=45.91.116.6:4975 , piece_id=SEP6DNRO6UNERV5BAEUKYN4HI64XEGDRMSFZOPLZF5MFA5SCO52Q, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.100101136 (409h2m56.971184171s) [noise=true , address=45.91.116.14:4190 , piece_id=C4KLRDIZUDGV42PR7IT6U566JWI4Y74GTQYIQWXDZD23OZXQJN3A, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.584867928 (409h2m57.455950963s) [noise=true , address=145.79.227.50:4374 , piece_id=KO26353DG3WVV6KCJK4MSMEPMWQHMDJ7R4UH4DWIDQMVO3WX2OQA, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.604114867 (409h2m57.475197902s) [noise=true , address=145.79.227.39:4015 , piece_id=YUWO5NRBGZAYMOAULJECX7EBTKAH5UZU3IQWW7JJUTJFLHRIXD7Q, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.608206619 (409h2m57.479289654s) [noise=true , address=145.79.227.43:4157 , piece_id=IAMJUVMZXDJB23XYKC3W777NVQ7AK7V66TRYTU5T6FFH53DI6TOA, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.608293174 (409h2m57.479376219s) [noise=true , address=145.79.227.9:4366 , piece_id=SIWOUBIJIX2KK5ZZSS7M4LDQ3PBPUWUXV4MYPNUAP6DBCCUSH4UQ, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.631338686 (409h2m57.502421721s) [noise=true , address=145.79.227.33:4871 , piece_id=LETKGEFTDBTLOJ2AUG2R6YLM2EKXFNNBVONGR5GQ7ROAQ3MJFOFA, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.675430132 (409h2m57.546513167s) [noise=true , address=145.79.227.16:4911 , piece_id=TBC4S2PNDODYPZLRWSMI3EOVBLPSP2W5CKI7SHLCSV2ENXSRCEZQ, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.883038140 (409h2m57.754121175s) [noise=true , address=145.79.227.48:4301 , piece_id=PELYPBNNTMBONPHJEGXPBROIQCSQFTVDH5NGFZ2O6BFYPFWN2XKA, seg_pos={0 0}, upload=359102]: Done uploading piece. err: <nil> Dec 22 16:31:57.883057828 (409h2m57.754140873s) [seg_pos={0 0}, upload=359102]: Segment reached optimal threshold of 13 pieces. Dec 22 16:31:57.883249703 (409h2m57.754332738s) [noise=true , address=145.79.227.37:4566 , piece_id=VDFSVSQF5K3OEFHHROWBXLQVWCOPJDDJMVTXOJPGRIVCVEALHOUQ, seg_pos={0 0}, upload=359102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 22 16:31:57.883255545 (409h2m57.7543386s) [noise=true , address=145.79.227.37:4566 , piece_id=VDFSVSQF5K3OEFHHROWBXLQVWCOPJDDJMVTXOJPGRIVCVEALHOUQ, seg_pos={0 0}, upload=359102]: Long tail cancellation due to optimal threshold Dec 22 16:31:57.883280411 (409h2m57.754363446s) [noise=true , address=145.79.227.20:4438 , piece_id=DW43VK3SXFFF2SHNEX62RP65EQNSVNEAFWGHZTYCFIAGZLHNXSWQ, seg_pos={0 0}, upload=359102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 22 16:31:57.883288908 (409h2m57.754371953s) [noise=true , address=145.79.227.20:4438 , piece_id=DW43VK3SXFFF2SHNEX62RP65EQNSVNEAFWGHZTYCFIAGZLHNXSWQ, seg_pos={0 0}, upload=359102]: Long tail cancellation due to optimal threshold Dec 22 16:31:57.883366205 (409h2m57.75444924s) [noise=true , address=145.79.227.5:4221 , piece_id=BRFZQIFCO5H23N7WRQCMALDQJYIU2RVTKZ5QPR4FMXYXNZ7AIY7A, seg_pos={0 0}, upload=359102]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:57766->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:57766->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57766->145.79.227.5:4221: use of closed network connection Dec 22 16:31:57.883371866 (409h2m57.754454911s) [noise=true , address=145.79.227.5:4221 , piece_id=BRFZQIFCO5H23N7WRQCMALDQJYIU2RVTKZ5QPR4FMXYXNZ7AIY7A, seg_pos={0 0}, upload=359102]: Long tail cancellation due to optimal threshold Dec 22 16:31:57.883392114 (409h2m57.754475149s) [seg_pos={0 0}, upload=359102]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:31:57.883473209 (409h2m57.754556244s) [upload=359102]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:31:57.883480342 (409h2m57.754563377s) [upload=359102]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:31:58.277921430 (409h2m58.149004465s) [upload=359102]: Done uploading segments... Dec 22 16:31:58.277938593 (409h2m58.149021628s) [upload=359102]: Upload finished. err: <nil> Duration: 30.004813937s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.405940002s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:32:22 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.405969237s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.449551546s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:31:49 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.449578187s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.025346431s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:31:47 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 6.025375116s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 477.074385ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:31:44 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 477.110775ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X....X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..XX.X...] duration: 7.852619481s trace: https://jaeger.dev.storj.io/trace/172e4e91afed93b2 FAIL (2025-12-22 16:31:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 16:29:52.088226640 (409h0m51.959309675s) [upload=359068]: Starting upload Dec 22 16:29:52.088281764 (409h0m51.9593648s) [upload=359068]: Done starting upload Dec 22 16:29:52.088403276 (409h0m51.959486311s) [upload=359068]: Uploading segments... Dec 22 16:29:52.088490863 (409h0m51.959573898s) [upload=359068]: Got next segment. Inline: false Dec 22 16:29:52.088520740 (409h0m51.959603775s) [upload=359068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:29:52.088526130 (409h0m51.959609165s) [upload=359068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:29:52.423917829 (409h0m52.295000864s) [seg_pos={0 0}, upload=359068]: Begin upload segment... Dec 22 16:29:52.423961392 (409h0m52.295044438s) [seg_pos={0 0}, upload=359068]: Done begin upload segment. Dec 22 16:29:52.423991730 (409h0m52.295074765s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424000477 (409h0m52.295083512s) [noise=true , address=145.79.227.41:4085 , piece_id=ZPEFMQ5XPUGPXZ7YZAVCF7EOMS2Z2K6M7PQ6G3K52LH6IJDANAWQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424087983 (409h0m52.295171028s) [noise=true , address=145.79.227.47:4266 , piece_id=FJQMQCB3WHNR4L7FICOX4GZ7NRSDKBWWKV4LYW5XCDUDYIXWL5ZA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424400789 (409h0m52.295483824s) [noise=true , address=145.79.227.35:4482 , piece_id=JGT73MN5OO4A5GGRZANHPDIYJLAS5BD33M2KIZFASWLGJOCHUJRQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424669921 (409h0m52.295752956s) [noise=true , address=145.79.227.13:4803 , piece_id=LVWDADURP4ITQIWJADVNAGYIEB7AMEAI2OUEOH222DAW526MJ42A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424957819 (409h0m52.296040854s) [noise=true , address=45.91.116.12:4228 , piece_id=M227OP4BG35OBWIAOBUS4KOKXEZU2UZXATFABZWHWBGKCBPBC7AQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.425320340 (409h0m52.296403375s) [noise=true , address=145.79.227.42:4121 , piece_id=N43G3RPZAP3YKNXECKNQRC3ZO7HFZHQAKP3HEGMFOHMVFY64VKRA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.425633947 (409h0m52.296716992s) [noise=true , address=145.79.227.50:4375 , piece_id=JZ4N3JQ5HEJPGSXNOIFZOJCUV2UHBUTICN37M2NI7GRTLHJM4HIQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.426917351 (409h0m52.298000396s) [noise=true , address=145.79.227.19:4078 , piece_id=GHWGHQH2KPYC54UABIYCXUX6YF33VNPEZZEDLAUPDIYK3T2XOC3A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.427302585 (409h0m52.29838562s) [noise=true , address=145.79.227.20:4438 , piece_id=D6IZPQYTVGQBVTFXYMYZBIN4ABJFHNODDOFN4DD54R5IAXIBTQ5A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428160949 (409h0m52.299243984s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428305253 (409h0m52.299388288s) [noise=true , address=77.90.31.16:4736 , piece_id=LOCX4A5WWYJH6BQWHNU7KJE37DC7SLIJGFGB7G7KCTYBHNXFU42A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428688383 (409h0m52.299771419s) [noise=true , address=77.90.31.19:4842 , piece_id=LXVH4V2QEYSJ4DJDDVIZFK5DMGCCVRHCTJTCPNXYGWX6TDTR4O7Q, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429000026 (409h0m52.300083061s) [noise=true , address=45.91.116.5:4297 , piece_id=XYEDN5WIDLPFH2ZVJP236HOY6PLRWVVZXPKJOME3IDYOAIWGKXBA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429045253 (409h0m52.300128288s) [noise=true , address=145.79.227.8:4329 , piece_id=TFCUJV7VACVHEJLP5TCJS4ZX5ZAVI6BYCAIJVNN5BWESUTHSU67A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429429445 (409h0m52.30051248s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:54.311323870 (409h0m54.182406905s) [upload=359068]: Next returned nil segment Dec 22 16:29:54.311344809 (409h0m54.182427854s) [upload=359068]: Waiting for error group managing segments... Dec 22 16:29:54.612256613 (409h0m54.483339648s) [noise=true , address=77.90.31.16:4736 , piece_id=LOCX4A5WWYJH6BQWHNU7KJE37DC7SLIJGFGB7G7KCTYBHNXFU42A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.721093255 (409h0m54.59217629s) [noise=true , address=45.91.116.5:4297 , piece_id=XYEDN5WIDLPFH2ZVJP236HOY6PLRWVVZXPKJOME3IDYOAIWGKXBA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.810681315 (409h0m54.68176437s) [noise=true , address=77.90.31.19:4842 , piece_id=LXVH4V2QEYSJ4DJDDVIZFK5DMGCCVRHCTJTCPNXYGWX6TDTR4O7Q, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.996224158 (409h0m54.867307193s) [noise=true , address=45.91.116.12:4228 , piece_id=M227OP4BG35OBWIAOBUS4KOKXEZU2UZXATFABZWHWBGKCBPBC7AQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.079980774 (409h0m54.951063819s) [noise=true , address=145.79.227.13:4803 , piece_id=LVWDADURP4ITQIWJADVNAGYIEB7AMEAI2OUEOH222DAW526MJ42A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.094779964 (409h0m54.965862999s) [noise=true , address=145.79.227.8:4329 , piece_id=TFCUJV7VACVHEJLP5TCJS4ZX5ZAVI6BYCAIJVNN5BWESUTHSU67A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.098327640 (409h0m54.969410675s) [noise=true , address=145.79.227.47:4266 , piece_id=FJQMQCB3WHNR4L7FICOX4GZ7NRSDKBWWKV4LYW5XCDUDYIXWL5ZA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.107965321 (409h0m54.979048366s) [noise=true , address=145.79.227.35:4482 , piece_id=JGT73MN5OO4A5GGRZANHPDIYJLAS5BD33M2KIZFASWLGJOCHUJRQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.115118572 (409h0m54.986201607s) [noise=true , address=145.79.227.41:4085 , piece_id=ZPEFMQ5XPUGPXZ7YZAVCF7EOMS2Z2K6M7PQ6G3K52LH6IJDANAWQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.292535302 (409h0m55.163618357s) [noise=true , address=145.79.227.50:4375 , piece_id=JZ4N3JQ5HEJPGSXNOIFZOJCUV2UHBUTICN37M2NI7GRTLHJM4HIQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.301194139 (409h0m55.172277184s) [noise=true , address=145.79.227.42:4121 , piece_id=N43G3RPZAP3YKNXECKNQRC3ZO7HFZHQAKP3HEGMFOHMVFY64VKRA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.322278207 (409h0m55.193361242s) [noise=true , address=145.79.227.20:4438 , piece_id=D6IZPQYTVGQBVTFXYMYZBIN4ABJFHNODDOFN4DD54R5IAXIBTQ5A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.326330362 (409h0m55.197413408s) [noise=true , address=145.79.227.19:4078 , piece_id=GHWGHQH2KPYC54UABIYCXUX6YF33VNPEZZEDLAUPDIYK3T2XOC3A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.326349099 (409h0m55.197432134s) [seg_pos={0 0}, upload=359068]: Segment reached optimal threshold of 13 pieces. Dec 22 16:29:55.326530724 (409h0m55.197613759s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326536725 (409h0m55.197619771s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326650201 (409h0m55.197733236s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326654750 (409h0m55.197737785s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326925826 (409h0m55.198008871s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326945354 (409h0m55.198028399s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326974379 (409h0m55.198057414s) [seg_pos={0 0}, upload=359068]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:29:55.327107542 (409h0m55.198190587s) [upload=359068]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:29:55.327115788 (409h0m55.198198834s) [upload=359068]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:29:55.758862422 (409h0m55.629945467s) [upload=359068]: Done uploading segments... Dec 22 16:29:55.758885515 (409h0m55.62996855s) [upload=359068]: Upload finished. err: <nil> Duration: 7.852649278s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.302386143s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:31:25 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.302413966s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.387461517s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:31:19 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.387487536s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX....X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..XX.X..] duration: 8.66596476s trace: https://jaeger.dev.storj.io/trace/172e4e91afed93b2 FAIL (2025-12-22 16:31:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 16:29:52.088226640 (409h0m51.959309675s) [upload=359068]: Starting upload Dec 22 16:29:52.088281764 (409h0m51.9593648s) [upload=359068]: Done starting upload Dec 22 16:29:52.088403276 (409h0m51.959486311s) [upload=359068]: Uploading segments... Dec 22 16:29:52.088490863 (409h0m51.959573898s) [upload=359068]: Got next segment. Inline: false Dec 22 16:29:52.088520740 (409h0m51.959603775s) [upload=359068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:29:52.088526130 (409h0m51.959609165s) [upload=359068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:29:52.423917829 (409h0m52.295000864s) [seg_pos={0 0}, upload=359068]: Begin upload segment... Dec 22 16:29:52.423961392 (409h0m52.295044438s) [seg_pos={0 0}, upload=359068]: Done begin upload segment. Dec 22 16:29:52.423991730 (409h0m52.295074765s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424000477 (409h0m52.295083512s) [noise=true , address=145.79.227.41:4085 , piece_id=ZPEFMQ5XPUGPXZ7YZAVCF7EOMS2Z2K6M7PQ6G3K52LH6IJDANAWQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424087983 (409h0m52.295171028s) [noise=true , address=145.79.227.47:4266 , piece_id=FJQMQCB3WHNR4L7FICOX4GZ7NRSDKBWWKV4LYW5XCDUDYIXWL5ZA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424400789 (409h0m52.295483824s) [noise=true , address=145.79.227.35:4482 , piece_id=JGT73MN5OO4A5GGRZANHPDIYJLAS5BD33M2KIZFASWLGJOCHUJRQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424669921 (409h0m52.295752956s) [noise=true , address=145.79.227.13:4803 , piece_id=LVWDADURP4ITQIWJADVNAGYIEB7AMEAI2OUEOH222DAW526MJ42A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424957819 (409h0m52.296040854s) [noise=true , address=45.91.116.12:4228 , piece_id=M227OP4BG35OBWIAOBUS4KOKXEZU2UZXATFABZWHWBGKCBPBC7AQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.425320340 (409h0m52.296403375s) [noise=true , address=145.79.227.42:4121 , piece_id=N43G3RPZAP3YKNXECKNQRC3ZO7HFZHQAKP3HEGMFOHMVFY64VKRA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.425633947 (409h0m52.296716992s) [noise=true , address=145.79.227.50:4375 , piece_id=JZ4N3JQ5HEJPGSXNOIFZOJCUV2UHBUTICN37M2NI7GRTLHJM4HIQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.426917351 (409h0m52.298000396s) [noise=true , address=145.79.227.19:4078 , piece_id=GHWGHQH2KPYC54UABIYCXUX6YF33VNPEZZEDLAUPDIYK3T2XOC3A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.427302585 (409h0m52.29838562s) [noise=true , address=145.79.227.20:4438 , piece_id=D6IZPQYTVGQBVTFXYMYZBIN4ABJFHNODDOFN4DD54R5IAXIBTQ5A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428160949 (409h0m52.299243984s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428305253 (409h0m52.299388288s) [noise=true , address=77.90.31.16:4736 , piece_id=LOCX4A5WWYJH6BQWHNU7KJE37DC7SLIJGFGB7G7KCTYBHNXFU42A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428688383 (409h0m52.299771419s) [noise=true , address=77.90.31.19:4842 , piece_id=LXVH4V2QEYSJ4DJDDVIZFK5DMGCCVRHCTJTCPNXYGWX6TDTR4O7Q, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429000026 (409h0m52.300083061s) [noise=true , address=45.91.116.5:4297 , piece_id=XYEDN5WIDLPFH2ZVJP236HOY6PLRWVVZXPKJOME3IDYOAIWGKXBA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429045253 (409h0m52.300128288s) [noise=true , address=145.79.227.8:4329 , piece_id=TFCUJV7VACVHEJLP5TCJS4ZX5ZAVI6BYCAIJVNN5BWESUTHSU67A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429429445 (409h0m52.30051248s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:54.311323870 (409h0m54.182406905s) [upload=359068]: Next returned nil segment Dec 22 16:29:54.311344809 (409h0m54.182427854s) [upload=359068]: Waiting for error group managing segments... Dec 22 16:29:54.612256613 (409h0m54.483339648s) [noise=true , address=77.90.31.16:4736 , piece_id=LOCX4A5WWYJH6BQWHNU7KJE37DC7SLIJGFGB7G7KCTYBHNXFU42A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.721093255 (409h0m54.59217629s) [noise=true , address=45.91.116.5:4297 , piece_id=XYEDN5WIDLPFH2ZVJP236HOY6PLRWVVZXPKJOME3IDYOAIWGKXBA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.810681315 (409h0m54.68176437s) [noise=true , address=77.90.31.19:4842 , piece_id=LXVH4V2QEYSJ4DJDDVIZFK5DMGCCVRHCTJTCPNXYGWX6TDTR4O7Q, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.996224158 (409h0m54.867307193s) [noise=true , address=45.91.116.12:4228 , piece_id=M227OP4BG35OBWIAOBUS4KOKXEZU2UZXATFABZWHWBGKCBPBC7AQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.079980774 (409h0m54.951063819s) [noise=true , address=145.79.227.13:4803 , piece_id=LVWDADURP4ITQIWJADVNAGYIEB7AMEAI2OUEOH222DAW526MJ42A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.094779964 (409h0m54.965862999s) [noise=true , address=145.79.227.8:4329 , piece_id=TFCUJV7VACVHEJLP5TCJS4ZX5ZAVI6BYCAIJVNN5BWESUTHSU67A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.098327640 (409h0m54.969410675s) [noise=true , address=145.79.227.47:4266 , piece_id=FJQMQCB3WHNR4L7FICOX4GZ7NRSDKBWWKV4LYW5XCDUDYIXWL5ZA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.107965321 (409h0m54.979048366s) [noise=true , address=145.79.227.35:4482 , piece_id=JGT73MN5OO4A5GGRZANHPDIYJLAS5BD33M2KIZFASWLGJOCHUJRQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.115118572 (409h0m54.986201607s) [noise=true , address=145.79.227.41:4085 , piece_id=ZPEFMQ5XPUGPXZ7YZAVCF7EOMS2Z2K6M7PQ6G3K52LH6IJDANAWQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.292535302 (409h0m55.163618357s) [noise=true , address=145.79.227.50:4375 , piece_id=JZ4N3JQ5HEJPGSXNOIFZOJCUV2UHBUTICN37M2NI7GRTLHJM4HIQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.301194139 (409h0m55.172277184s) [noise=true , address=145.79.227.42:4121 , piece_id=N43G3RPZAP3YKNXECKNQRC3ZO7HFZHQAKP3HEGMFOHMVFY64VKRA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.322278207 (409h0m55.193361242s) [noise=true , address=145.79.227.20:4438 , piece_id=D6IZPQYTVGQBVTFXYMYZBIN4ABJFHNODDOFN4DD54R5IAXIBTQ5A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.326330362 (409h0m55.197413408s) [noise=true , address=145.79.227.19:4078 , piece_id=GHWGHQH2KPYC54UABIYCXUX6YF33VNPEZZEDLAUPDIYK3T2XOC3A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.326349099 (409h0m55.197432134s) [seg_pos={0 0}, upload=359068]: Segment reached optimal threshold of 13 pieces. Dec 22 16:29:55.326530724 (409h0m55.197613759s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326536725 (409h0m55.197619771s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326650201 (409h0m55.197733236s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326654750 (409h0m55.197737785s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326925826 (409h0m55.198008871s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326945354 (409h0m55.198028399s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326974379 (409h0m55.198057414s) [seg_pos={0 0}, upload=359068]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:29:55.327107542 (409h0m55.198190587s) [upload=359068]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:29:55.327115788 (409h0m55.198198834s) [upload=359068]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:29:55.758862422 (409h0m55.629945467s) [upload=359068]: Done uploading segments... Dec 22 16:29:55.758885515 (409h0m55.62996855s) [upload=359068]: Upload finished. err: <nil> Duration: 8.665991781s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 469.164765ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:30:42 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 469.194972ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X............................X..............................] duration: 5.28535854s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:30:40 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.285385661s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXX....X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..XX.X.] duration: 8.449137978s trace: https://jaeger.dev.storj.io/trace/172e4e91afed93b2 FAIL (2025-12-22 16:30:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 16:29:52.088226640 (409h0m51.959309675s) [upload=359068]: Starting upload Dec 22 16:29:52.088281764 (409h0m51.9593648s) [upload=359068]: Done starting upload Dec 22 16:29:52.088403276 (409h0m51.959486311s) [upload=359068]: Uploading segments... Dec 22 16:29:52.088490863 (409h0m51.959573898s) [upload=359068]: Got next segment. Inline: false Dec 22 16:29:52.088520740 (409h0m51.959603775s) [upload=359068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:29:52.088526130 (409h0m51.959609165s) [upload=359068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:29:52.423917829 (409h0m52.295000864s) [seg_pos={0 0}, upload=359068]: Begin upload segment... Dec 22 16:29:52.423961392 (409h0m52.295044438s) [seg_pos={0 0}, upload=359068]: Done begin upload segment. Dec 22 16:29:52.423991730 (409h0m52.295074765s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424000477 (409h0m52.295083512s) [noise=true , address=145.79.227.41:4085 , piece_id=ZPEFMQ5XPUGPXZ7YZAVCF7EOMS2Z2K6M7PQ6G3K52LH6IJDANAWQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424087983 (409h0m52.295171028s) [noise=true , address=145.79.227.47:4266 , piece_id=FJQMQCB3WHNR4L7FICOX4GZ7NRSDKBWWKV4LYW5XCDUDYIXWL5ZA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424400789 (409h0m52.295483824s) [noise=true , address=145.79.227.35:4482 , piece_id=JGT73MN5OO4A5GGRZANHPDIYJLAS5BD33M2KIZFASWLGJOCHUJRQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424669921 (409h0m52.295752956s) [noise=true , address=145.79.227.13:4803 , piece_id=LVWDADURP4ITQIWJADVNAGYIEB7AMEAI2OUEOH222DAW526MJ42A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424957819 (409h0m52.296040854s) [noise=true , address=45.91.116.12:4228 , piece_id=M227OP4BG35OBWIAOBUS4KOKXEZU2UZXATFABZWHWBGKCBPBC7AQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.425320340 (409h0m52.296403375s) [noise=true , address=145.79.227.42:4121 , piece_id=N43G3RPZAP3YKNXECKNQRC3ZO7HFZHQAKP3HEGMFOHMVFY64VKRA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.425633947 (409h0m52.296716992s) [noise=true , address=145.79.227.50:4375 , piece_id=JZ4N3JQ5HEJPGSXNOIFZOJCUV2UHBUTICN37M2NI7GRTLHJM4HIQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.426917351 (409h0m52.298000396s) [noise=true , address=145.79.227.19:4078 , piece_id=GHWGHQH2KPYC54UABIYCXUX6YF33VNPEZZEDLAUPDIYK3T2XOC3A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.427302585 (409h0m52.29838562s) [noise=true , address=145.79.227.20:4438 , piece_id=D6IZPQYTVGQBVTFXYMYZBIN4ABJFHNODDOFN4DD54R5IAXIBTQ5A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428160949 (409h0m52.299243984s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428305253 (409h0m52.299388288s) [noise=true , address=77.90.31.16:4736 , piece_id=LOCX4A5WWYJH6BQWHNU7KJE37DC7SLIJGFGB7G7KCTYBHNXFU42A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428688383 (409h0m52.299771419s) [noise=true , address=77.90.31.19:4842 , piece_id=LXVH4V2QEYSJ4DJDDVIZFK5DMGCCVRHCTJTCPNXYGWX6TDTR4O7Q, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429000026 (409h0m52.300083061s) [noise=true , address=45.91.116.5:4297 , piece_id=XYEDN5WIDLPFH2ZVJP236HOY6PLRWVVZXPKJOME3IDYOAIWGKXBA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429045253 (409h0m52.300128288s) [noise=true , address=145.79.227.8:4329 , piece_id=TFCUJV7VACVHEJLP5TCJS4ZX5ZAVI6BYCAIJVNN5BWESUTHSU67A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429429445 (409h0m52.30051248s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:54.311323870 (409h0m54.182406905s) [upload=359068]: Next returned nil segment Dec 22 16:29:54.311344809 (409h0m54.182427854s) [upload=359068]: Waiting for error group managing segments... Dec 22 16:29:54.612256613 (409h0m54.483339648s) [noise=true , address=77.90.31.16:4736 , piece_id=LOCX4A5WWYJH6BQWHNU7KJE37DC7SLIJGFGB7G7KCTYBHNXFU42A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.721093255 (409h0m54.59217629s) [noise=true , address=45.91.116.5:4297 , piece_id=XYEDN5WIDLPFH2ZVJP236HOY6PLRWVVZXPKJOME3IDYOAIWGKXBA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.810681315 (409h0m54.68176437s) [noise=true , address=77.90.31.19:4842 , piece_id=LXVH4V2QEYSJ4DJDDVIZFK5DMGCCVRHCTJTCPNXYGWX6TDTR4O7Q, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.996224158 (409h0m54.867307193s) [noise=true , address=45.91.116.12:4228 , piece_id=M227OP4BG35OBWIAOBUS4KOKXEZU2UZXATFABZWHWBGKCBPBC7AQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.079980774 (409h0m54.951063819s) [noise=true , address=145.79.227.13:4803 , piece_id=LVWDADURP4ITQIWJADVNAGYIEB7AMEAI2OUEOH222DAW526MJ42A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.094779964 (409h0m54.965862999s) [noise=true , address=145.79.227.8:4329 , piece_id=TFCUJV7VACVHEJLP5TCJS4ZX5ZAVI6BYCAIJVNN5BWESUTHSU67A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.098327640 (409h0m54.969410675s) [noise=true , address=145.79.227.47:4266 , piece_id=FJQMQCB3WHNR4L7FICOX4GZ7NRSDKBWWKV4LYW5XCDUDYIXWL5ZA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.107965321 (409h0m54.979048366s) [noise=true , address=145.79.227.35:4482 , piece_id=JGT73MN5OO4A5GGRZANHPDIYJLAS5BD33M2KIZFASWLGJOCHUJRQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.115118572 (409h0m54.986201607s) [noise=true , address=145.79.227.41:4085 , piece_id=ZPEFMQ5XPUGPXZ7YZAVCF7EOMS2Z2K6M7PQ6G3K52LH6IJDANAWQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.292535302 (409h0m55.163618357s) [noise=true , address=145.79.227.50:4375 , piece_id=JZ4N3JQ5HEJPGSXNOIFZOJCUV2UHBUTICN37M2NI7GRTLHJM4HIQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.301194139 (409h0m55.172277184s) [noise=true , address=145.79.227.42:4121 , piece_id=N43G3RPZAP3YKNXECKNQRC3ZO7HFZHQAKP3HEGMFOHMVFY64VKRA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.322278207 (409h0m55.193361242s) [noise=true , address=145.79.227.20:4438 , piece_id=D6IZPQYTVGQBVTFXYMYZBIN4ABJFHNODDOFN4DD54R5IAXIBTQ5A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.326330362 (409h0m55.197413408s) [noise=true , address=145.79.227.19:4078 , piece_id=GHWGHQH2KPYC54UABIYCXUX6YF33VNPEZZEDLAUPDIYK3T2XOC3A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.326349099 (409h0m55.197432134s) [seg_pos={0 0}, upload=359068]: Segment reached optimal threshold of 13 pieces. Dec 22 16:29:55.326530724 (409h0m55.197613759s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326536725 (409h0m55.197619771s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326650201 (409h0m55.197733236s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326654750 (409h0m55.197737785s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326925826 (409h0m55.198008871s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326945354 (409h0m55.198028399s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326974379 (409h0m55.198057414s) [seg_pos={0 0}, upload=359068]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:29:55.327107542 (409h0m55.198190587s) [upload=359068]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:29:55.327115788 (409h0m55.198198834s) [upload=359068]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:29:55.758862422 (409h0m55.629945467s) [upload=359068]: Done uploading segments... Dec 22 16:29:55.758885515 (409h0m55.62996855s) [upload=359068]: Upload finished. err: <nil> Duration: 8.449168816s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.409439015s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:30:28 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.409468861s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XXXX....X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..XX.X] duration: 30.003600724s trace: https://jaeger.dev.storj.io/trace/172e4e91afed93b2 FAIL (2025-12-22 16:30:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 16:29:52.088226640 (409h0m51.959309675s) [upload=359068]: Starting upload Dec 22 16:29:52.088281764 (409h0m51.9593648s) [upload=359068]: Done starting upload Dec 22 16:29:52.088403276 (409h0m51.959486311s) [upload=359068]: Uploading segments... Dec 22 16:29:52.088490863 (409h0m51.959573898s) [upload=359068]: Got next segment. Inline: false Dec 22 16:29:52.088520740 (409h0m51.959603775s) [upload=359068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:29:52.088526130 (409h0m51.959609165s) [upload=359068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:29:52.423917829 (409h0m52.295000864s) [seg_pos={0 0}, upload=359068]: Begin upload segment... Dec 22 16:29:52.423961392 (409h0m52.295044438s) [seg_pos={0 0}, upload=359068]: Done begin upload segment. Dec 22 16:29:52.423991730 (409h0m52.295074765s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424000477 (409h0m52.295083512s) [noise=true , address=145.79.227.41:4085 , piece_id=ZPEFMQ5XPUGPXZ7YZAVCF7EOMS2Z2K6M7PQ6G3K52LH6IJDANAWQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424087983 (409h0m52.295171028s) [noise=true , address=145.79.227.47:4266 , piece_id=FJQMQCB3WHNR4L7FICOX4GZ7NRSDKBWWKV4LYW5XCDUDYIXWL5ZA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424400789 (409h0m52.295483824s) [noise=true , address=145.79.227.35:4482 , piece_id=JGT73MN5OO4A5GGRZANHPDIYJLAS5BD33M2KIZFASWLGJOCHUJRQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424669921 (409h0m52.295752956s) [noise=true , address=145.79.227.13:4803 , piece_id=LVWDADURP4ITQIWJADVNAGYIEB7AMEAI2OUEOH222DAW526MJ42A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.424957819 (409h0m52.296040854s) [noise=true , address=45.91.116.12:4228 , piece_id=M227OP4BG35OBWIAOBUS4KOKXEZU2UZXATFABZWHWBGKCBPBC7AQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.425320340 (409h0m52.296403375s) [noise=true , address=145.79.227.42:4121 , piece_id=N43G3RPZAP3YKNXECKNQRC3ZO7HFZHQAKP3HEGMFOHMVFY64VKRA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.425633947 (409h0m52.296716992s) [noise=true , address=145.79.227.50:4375 , piece_id=JZ4N3JQ5HEJPGSXNOIFZOJCUV2UHBUTICN37M2NI7GRTLHJM4HIQ, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.426917351 (409h0m52.298000396s) [noise=true , address=145.79.227.19:4078 , piece_id=GHWGHQH2KPYC54UABIYCXUX6YF33VNPEZZEDLAUPDIYK3T2XOC3A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.427302585 (409h0m52.29838562s) [noise=true , address=145.79.227.20:4438 , piece_id=D6IZPQYTVGQBVTFXYMYZBIN4ABJFHNODDOFN4DD54R5IAXIBTQ5A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428160949 (409h0m52.299243984s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428305253 (409h0m52.299388288s) [noise=true , address=77.90.31.16:4736 , piece_id=LOCX4A5WWYJH6BQWHNU7KJE37DC7SLIJGFGB7G7KCTYBHNXFU42A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.428688383 (409h0m52.299771419s) [noise=true , address=77.90.31.19:4842 , piece_id=LXVH4V2QEYSJ4DJDDVIZFK5DMGCCVRHCTJTCPNXYGWX6TDTR4O7Q, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429000026 (409h0m52.300083061s) [noise=true , address=45.91.116.5:4297 , piece_id=XYEDN5WIDLPFH2ZVJP236HOY6PLRWVVZXPKJOME3IDYOAIWGKXBA, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429045253 (409h0m52.300128288s) [noise=true , address=145.79.227.8:4329 , piece_id=TFCUJV7VACVHEJLP5TCJS4ZX5ZAVI6BYCAIJVNN5BWESUTHSU67A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:52.429429445 (409h0m52.30051248s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Uploading piece... Dec 22 16:29:54.311323870 (409h0m54.182406905s) [upload=359068]: Next returned nil segment Dec 22 16:29:54.311344809 (409h0m54.182427854s) [upload=359068]: Waiting for error group managing segments... Dec 22 16:29:54.612256613 (409h0m54.483339648s) [noise=true , address=77.90.31.16:4736 , piece_id=LOCX4A5WWYJH6BQWHNU7KJE37DC7SLIJGFGB7G7KCTYBHNXFU42A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.721093255 (409h0m54.59217629s) [noise=true , address=45.91.116.5:4297 , piece_id=XYEDN5WIDLPFH2ZVJP236HOY6PLRWVVZXPKJOME3IDYOAIWGKXBA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.810681315 (409h0m54.68176437s) [noise=true , address=77.90.31.19:4842 , piece_id=LXVH4V2QEYSJ4DJDDVIZFK5DMGCCVRHCTJTCPNXYGWX6TDTR4O7Q, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:54.996224158 (409h0m54.867307193s) [noise=true , address=45.91.116.12:4228 , piece_id=M227OP4BG35OBWIAOBUS4KOKXEZU2UZXATFABZWHWBGKCBPBC7AQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.079980774 (409h0m54.951063819s) [noise=true , address=145.79.227.13:4803 , piece_id=LVWDADURP4ITQIWJADVNAGYIEB7AMEAI2OUEOH222DAW526MJ42A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.094779964 (409h0m54.965862999s) [noise=true , address=145.79.227.8:4329 , piece_id=TFCUJV7VACVHEJLP5TCJS4ZX5ZAVI6BYCAIJVNN5BWESUTHSU67A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.098327640 (409h0m54.969410675s) [noise=true , address=145.79.227.47:4266 , piece_id=FJQMQCB3WHNR4L7FICOX4GZ7NRSDKBWWKV4LYW5XCDUDYIXWL5ZA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.107965321 (409h0m54.979048366s) [noise=true , address=145.79.227.35:4482 , piece_id=JGT73MN5OO4A5GGRZANHPDIYJLAS5BD33M2KIZFASWLGJOCHUJRQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.115118572 (409h0m54.986201607s) [noise=true , address=145.79.227.41:4085 , piece_id=ZPEFMQ5XPUGPXZ7YZAVCF7EOMS2Z2K6M7PQ6G3K52LH6IJDANAWQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.292535302 (409h0m55.163618357s) [noise=true , address=145.79.227.50:4375 , piece_id=JZ4N3JQ5HEJPGSXNOIFZOJCUV2UHBUTICN37M2NI7GRTLHJM4HIQ, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.301194139 (409h0m55.172277184s) [noise=true , address=145.79.227.42:4121 , piece_id=N43G3RPZAP3YKNXECKNQRC3ZO7HFZHQAKP3HEGMFOHMVFY64VKRA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.322278207 (409h0m55.193361242s) [noise=true , address=145.79.227.20:4438 , piece_id=D6IZPQYTVGQBVTFXYMYZBIN4ABJFHNODDOFN4DD54R5IAXIBTQ5A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.326330362 (409h0m55.197413408s) [noise=true , address=145.79.227.19:4078 , piece_id=GHWGHQH2KPYC54UABIYCXUX6YF33VNPEZZEDLAUPDIYK3T2XOC3A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: <nil> Dec 22 16:29:55.326349099 (409h0m55.197432134s) [seg_pos={0 0}, upload=359068]: Segment reached optimal threshold of 13 pieces. Dec 22 16:29:55.326530724 (409h0m55.197613759s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326536725 (409h0m55.197619771s) [noise=true , address=145.79.227.43:4157 , piece_id=MEC5YMV7Y6KTLZ2SBDMHSVLUHH3NDYA3KETHOROGYGMWWNOA4Y2A, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326650201 (409h0m55.197733236s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326654750 (409h0m55.197737785s) [noise=true , address=145.79.227.39:4013 , piece_id=FD6UF4LDS4GQDBBF2NOIHXEOQ6TUF2HVXDW5R3PJBBTPPQMT4IYA, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326925826 (409h0m55.198008871s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2): protocol: expected piece hash; context canceled; EOF Dec 22 16:29:55.326945354 (409h0m55.198028399s) [noise=true , address=145.79.227.15:4875 , piece_id=BO2ADKYHFAJLYPTVXLGCEMWYBO2RF736IDK54LXVLFZBPMZXTJWA, seg_pos={0 0}, upload=359068]: Long tail cancellation due to optimal threshold Dec 22 16:29:55.326974379 (409h0m55.198057414s) [seg_pos={0 0}, upload=359068]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:29:55.327107542 (409h0m55.198190587s) [upload=359068]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:29:55.327115788 (409h0m55.198198834s) [upload=359068]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:29:55.758862422 (409h0m55.629945467s) [upload=359068]: Done uploading segments... Dec 22 16:29:55.758885515 (409h0m55.62996855s) [upload=359068]: Upload finished. err: <nil> Duration: 30.003623326s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.404458402s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:30:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.404487227s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 467.811867ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:29:51 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 467.84046ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X............................X.............................] duration: 4.893468389s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:29:35 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 4.89348513s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XXXX....X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..XX.] duration: 10.116479031s trace: https://jaeger.dev.storj.io/trace/afaf10638bc8a5 FAIL (2025-12-22 16:29:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: YFOSG6X3FUKPAQDNUS46QEN6TKPVZU7T23DU26HEJAHGJLTQFI4Q) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CQSKYN6XKXE3ASKW7GH6VRJWBYCM2XWN7OBCNCTVQ4HKDH3ZRYJQ) context deadline exceeded ======LOGS====== Dec 22 16:28:35.812001443 (408h59m35.683084478s) [upload=359051]: Starting upload Dec 22 16:28:35.812059213 (408h59m35.683142248s) [upload=359051]: Done starting upload Dec 22 16:28:35.812263963 (408h59m35.683346998s) [upload=359051]: Uploading segments... Dec 22 16:28:35.812321292 (408h59m35.683404367s) [upload=359051]: Got next segment. Inline: false Dec 22 16:28:35.812358974 (408h59m35.683442019s) [upload=359051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:28:35.812365206 (408h59m35.683448251s) [upload=359051]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:28:36.165463431 (408h59m36.036546466s) [seg_pos={0 0}, upload=359051]: Begin upload segment... Dec 22 16:28:36.165519367 (408h59m36.036602403s) [seg_pos={0 0}, upload=359051]: Done begin upload segment. Dec 22 16:28:36.165560696 (408h59m36.036643731s) [noise=true , address=145.79.227.38:4977 , piece_id=UY4TXHP6MVSX5L77MYZDXUE34OAZHLCEDQFO6SGQXIN2XEONFY4A, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.165593088 (408h59m36.036676123s) [noise=true , address=45.91.116.8:4046 , piece_id=UOUD2GKLRWTKQZRAGFYNFVIPHZROYUEDHTONEKGMPGIPC76OSF5A, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.165957792 (408h59m36.037040827s) [noise=true , address=145.79.227.9:4366 , piece_id=FQVPTBM2QXP75XDUNPB534QUXZMUM5EJHME73GV4G2X6YQJ2SWPA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.166255559 (408h59m36.037338594s) [noise=true , address=145.79.227.5:4221 , piece_id=APBDNO5HES7F6TIMCA3WK4IUFJULOELJBOV2HBJH7RMVCJRRHFZA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.166525683 (408h59m36.037608718s) [noise=true , address=45.91.116.9:4262 , piece_id=LXZLGW2OB7ZTETME3ICJS5ZJB66NIWBDBZ46EXL3RMDENNUPKVKA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.166835082 (408h59m36.037918217s) [noise=true , address=145.79.227.39:4015 , piece_id=FYLJN3QH5SC5NIR4M7BZMWF4WMQXLKSCRFQV54GHKL573QZ7FLXA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.167093515 (408h59m36.03817656s) [noise=true , address=77.90.31.12:4589 , piece_id=CFOBDGS7MRGKC77PL72FTRG6CGLGSEKC6ZEMXIUQQUI6JKUO4W4A, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.167358008 (408h59m36.038441043s) [noise=true , address=145.79.227.52:4447 , piece_id=73A5JY7AKKUW3GFH4FTYIHGTCMMBIHU5F3NLOJVBOJMIP2SB6PSQ, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.167721691 (408h59m36.038804726s) [noise=true , address=145.79.227.10:4401 , piece_id=CB72VDTVIGXKML6S6J5XP5RRRUP6TKYYQBUFDPSTUCMSCUCF2XIQ, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.168990526 (408h59m36.040073571s) [noise=true , address=77.90.31.19:4842 , piece_id=F3HBHRAXI5NGMDJG45SMQLSXTMOSYF6LKONDS2EGDQSPUYVKC46A, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.170311532 (408h59m36.041394577s) [noise=true , address=77.90.31.11:4556 , piece_id=YFOSG6X3FUKPAQDNUS46QEN6TKPVZU7T23DU26HEJAHGJLTQFI4Q, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.171517168 (408h59m36.042600203s) [noise=true , address=77.90.31.16:4736 , piece_id=Z5EJU25C6OEOAC3C2ZMCZ73FLPU6JRGFFMYHMGNCEO6G7KSJXHUA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.172630248 (408h59m36.043713283s) [noise=true , address=45.91.116.10:4081 , piece_id=CQSKYN6XKXE3ASKW7GH6VRJWBYCM2XWN7OBCNCTVQ4HKDH3ZRYJQ, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.173925594 (408h59m36.045008629s) [noise=true , address=45.91.116.11:4120 , piece_id=IJ4DEJNJ2GURQF3T5XA6ZXXQLNPRH7QBMWCW54JFWWJBKFIIZTBA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.175041128 (408h59m36.046124163s) [noise=true , address=145.79.227.20:4438 , piece_id=Z7Z3GJIH5U7JM5NFXW24KVDXNEVOWQZ3GY6423OPGC3DATWV6GCA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.176343898 (408h59m36.047426933s) [noise=true , address=45.91.116.6:4975 , piece_id=XXE77DC5TGSPEZIDDEPNB4UHYI7JGVSRNCPMQI7O4YBVWL6AKCOA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:37.323539651 (408h59m37.194622686s) [noise=true , address=77.90.31.12:4589 , piece_id=CFOBDGS7MRGKC77PL72FTRG6CGLGSEKC6ZEMXIUQQUI6JKUO4W4A, seg_pos={0 0}, upload=359051]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 22 16:28:38.058933928 (408h59m37.930016963s) [upload=359051]: Next returned nil segment Dec 22 16:28:38.058948837 (408h59m37.930031882s) [upload=359051]: Waiting for error group managing segments... Dec 22 16:28:38.393679457 (408h59m38.264762492s) [noise=true , address=77.90.31.19:4842 , piece_id=F3HBHRAXI5NGMDJG45SMQLSXTMOSYF6LKONDS2EGDQSPUYVKC46A, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.435918947 (408h59m38.307001982s) [noise=true , address=45.91.116.8:4046 , piece_id=UOUD2GKLRWTKQZRAGFYNFVIPHZROYUEDHTONEKGMPGIPC76OSF5A, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.469142901 (408h59m38.340225946s) [noise=true , address=45.91.116.10:4081 , piece_id=CQSKYN6XKXE3ASKW7GH6VRJWBYCM2XWN7OBCNCTVQ4HKDH3ZRYJQ, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.480038468 (408h59m38.351121503s) [noise=true , address=45.91.116.6:4975 , piece_id=XXE77DC5TGSPEZIDDEPNB4UHYI7JGVSRNCPMQI7O4YBVWL6AKCOA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.495720900 (408h59m38.366803945s) [noise=true , address=77.90.31.11:4556 , piece_id=YFOSG6X3FUKPAQDNUS46QEN6TKPVZU7T23DU26HEJAHGJLTQFI4Q, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.520490235 (408h59m38.39157327s) [noise=true , address=77.90.31.16:4736 , piece_id=Z5EJU25C6OEOAC3C2ZMCZ73FLPU6JRGFFMYHMGNCEO6G7KSJXHUA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.731918258 (408h59m38.603001293s) [noise=true , address=45.91.116.9:4262 , piece_id=LXZLGW2OB7ZTETME3ICJS5ZJB66NIWBDBZ46EXL3RMDENNUPKVKA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.059878596 (408h59m38.930961631s) [noise=true , address=145.79.227.5:4221 , piece_id=APBDNO5HES7F6TIMCA3WK4IUFJULOELJBOV2HBJH7RMVCJRRHFZA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.120519185 (408h59m38.99160224s) [noise=true , address=145.79.227.52:4447 , piece_id=73A5JY7AKKUW3GFH4FTYIHGTCMMBIHU5F3NLOJVBOJMIP2SB6PSQ, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.130571977 (408h59m39.001655012s) [noise=true , address=145.79.227.9:4366 , piece_id=FQVPTBM2QXP75XDUNPB534QUXZMUM5EJHME73GV4G2X6YQJ2SWPA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.144319064 (408h59m39.015402099s) [noise=true , address=145.79.227.38:4977 , piece_id=UY4TXHP6MVSX5L77MYZDXUE34OAZHLCEDQFO6SGQXIN2XEONFY4A, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.192087993 (408h59m39.063171028s) [noise=true , address=145.79.227.10:4401 , piece_id=CB72VDTVIGXKML6S6J5XP5RRRUP6TKYYQBUFDPSTUCMSCUCF2XIQ, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.247130523 (408h59m39.118213558s) [noise=true , address=145.79.227.39:4015 , piece_id=FYLJN3QH5SC5NIR4M7BZMWF4WMQXLKSCRFQV54GHKL573QZ7FLXA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.247151553 (408h59m39.118234588s) [seg_pos={0 0}, upload=359051]: Segment reached optimal threshold of 13 pieces. Dec 22 16:28:39.247340253 (408h59m39.118423288s) [noise=true , address=45.91.116.11:4120 , piece_id=IJ4DEJNJ2GURQF3T5XA6ZXXQLNPRH7QBMWCW54JFWWJBKFIIZTBA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 16:28:39.247347166 (408h59m39.118430201s) [noise=true , address=45.91.116.11:4120 , piece_id=IJ4DEJNJ2GURQF3T5XA6ZXXQLNPRH7QBMWCW54JFWWJBKFIIZTBA, seg_pos={0 0}, upload=359051]: Long tail cancellation due to optimal threshold Dec 22 16:28:39.247744692 (408h59m39.118827737s) [noise=true , address=145.79.227.20:4438 , piece_id=Z7Z3GJIH5U7JM5NFXW24KVDXNEVOWQZ3GY6423OPGC3DATWV6GCA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 22 16:28:39.247781953 (408h59m39.118865008s) [noise=true , address=145.79.227.20:4438 , piece_id=Z7Z3GJIH5U7JM5NFXW24KVDXNEVOWQZ3GY6423OPGC3DATWV6GCA, seg_pos={0 0}, upload=359051]: Long tail cancellation due to optimal threshold Dec 22 16:28:39.247810608 (408h59m39.118893653s) [seg_pos={0 0}, upload=359051]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:28:39.247971094 (408h59m39.119054139s) [upload=359051]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:28:39.247978087 (408h59m39.119061122s) [upload=359051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:28:39.671766327 (408h59m39.542849372s) [upload=359051]: Done uploading segments... Dec 22 16:28:39.671799079 (408h59m39.542882124s) [upload=359051]: Upload finished. err: <nil> Duration: 10.116505883s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X............................X............................] duration: 5.913681686s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:29:05 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.913712915s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XXXX....X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..XX] duration: 30.003523733s trace: https://jaeger.dev.storj.io/trace/afaf10638bc8a5 FAIL (2025-12-22 16:29:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: YFOSG6X3FUKPAQDNUS46QEN6TKPVZU7T23DU26HEJAHGJLTQFI4Q) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CQSKYN6XKXE3ASKW7GH6VRJWBYCM2XWN7OBCNCTVQ4HKDH3ZRYJQ) context deadline exceeded ======LOGS====== Dec 22 16:28:35.812001443 (408h59m35.683084478s) [upload=359051]: Starting upload Dec 22 16:28:35.812059213 (408h59m35.683142248s) [upload=359051]: Done starting upload Dec 22 16:28:35.812263963 (408h59m35.683346998s) [upload=359051]: Uploading segments... Dec 22 16:28:35.812321292 (408h59m35.683404367s) [upload=359051]: Got next segment. Inline: false Dec 22 16:28:35.812358974 (408h59m35.683442019s) [upload=359051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:28:35.812365206 (408h59m35.683448251s) [upload=359051]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:28:36.165463431 (408h59m36.036546466s) [seg_pos={0 0}, upload=359051]: Begin upload segment... Dec 22 16:28:36.165519367 (408h59m36.036602403s) [seg_pos={0 0}, upload=359051]: Done begin upload segment. Dec 22 16:28:36.165560696 (408h59m36.036643731s) [noise=true , address=145.79.227.38:4977 , piece_id=UY4TXHP6MVSX5L77MYZDXUE34OAZHLCEDQFO6SGQXIN2XEONFY4A, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.165593088 (408h59m36.036676123s) [noise=true , address=45.91.116.8:4046 , piece_id=UOUD2GKLRWTKQZRAGFYNFVIPHZROYUEDHTONEKGMPGIPC76OSF5A, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.165957792 (408h59m36.037040827s) [noise=true , address=145.79.227.9:4366 , piece_id=FQVPTBM2QXP75XDUNPB534QUXZMUM5EJHME73GV4G2X6YQJ2SWPA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.166255559 (408h59m36.037338594s) [noise=true , address=145.79.227.5:4221 , piece_id=APBDNO5HES7F6TIMCA3WK4IUFJULOELJBOV2HBJH7RMVCJRRHFZA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.166525683 (408h59m36.037608718s) [noise=true , address=45.91.116.9:4262 , piece_id=LXZLGW2OB7ZTETME3ICJS5ZJB66NIWBDBZ46EXL3RMDENNUPKVKA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.166835082 (408h59m36.037918217s) [noise=true , address=145.79.227.39:4015 , piece_id=FYLJN3QH5SC5NIR4M7BZMWF4WMQXLKSCRFQV54GHKL573QZ7FLXA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.167093515 (408h59m36.03817656s) [noise=true , address=77.90.31.12:4589 , piece_id=CFOBDGS7MRGKC77PL72FTRG6CGLGSEKC6ZEMXIUQQUI6JKUO4W4A, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.167358008 (408h59m36.038441043s) [noise=true , address=145.79.227.52:4447 , piece_id=73A5JY7AKKUW3GFH4FTYIHGTCMMBIHU5F3NLOJVBOJMIP2SB6PSQ, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.167721691 (408h59m36.038804726s) [noise=true , address=145.79.227.10:4401 , piece_id=CB72VDTVIGXKML6S6J5XP5RRRUP6TKYYQBUFDPSTUCMSCUCF2XIQ, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.168990526 (408h59m36.040073571s) [noise=true , address=77.90.31.19:4842 , piece_id=F3HBHRAXI5NGMDJG45SMQLSXTMOSYF6LKONDS2EGDQSPUYVKC46A, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.170311532 (408h59m36.041394577s) [noise=true , address=77.90.31.11:4556 , piece_id=YFOSG6X3FUKPAQDNUS46QEN6TKPVZU7T23DU26HEJAHGJLTQFI4Q, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.171517168 (408h59m36.042600203s) [noise=true , address=77.90.31.16:4736 , piece_id=Z5EJU25C6OEOAC3C2ZMCZ73FLPU6JRGFFMYHMGNCEO6G7KSJXHUA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.172630248 (408h59m36.043713283s) [noise=true , address=45.91.116.10:4081 , piece_id=CQSKYN6XKXE3ASKW7GH6VRJWBYCM2XWN7OBCNCTVQ4HKDH3ZRYJQ, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.173925594 (408h59m36.045008629s) [noise=true , address=45.91.116.11:4120 , piece_id=IJ4DEJNJ2GURQF3T5XA6ZXXQLNPRH7QBMWCW54JFWWJBKFIIZTBA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.175041128 (408h59m36.046124163s) [noise=true , address=145.79.227.20:4438 , piece_id=Z7Z3GJIH5U7JM5NFXW24KVDXNEVOWQZ3GY6423OPGC3DATWV6GCA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:36.176343898 (408h59m36.047426933s) [noise=true , address=45.91.116.6:4975 , piece_id=XXE77DC5TGSPEZIDDEPNB4UHYI7JGVSRNCPMQI7O4YBVWL6AKCOA, seg_pos={0 0}, upload=359051]: Uploading piece... Dec 22 16:28:37.323539651 (408h59m37.194622686s) [noise=true , address=77.90.31.12:4589 , piece_id=CFOBDGS7MRGKC77PL72FTRG6CGLGSEKC6ZEMXIUQQUI6JKUO4W4A, seg_pos={0 0}, upload=359051]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 22 16:28:38.058933928 (408h59m37.930016963s) [upload=359051]: Next returned nil segment Dec 22 16:28:38.058948837 (408h59m37.930031882s) [upload=359051]: Waiting for error group managing segments... Dec 22 16:28:38.393679457 (408h59m38.264762492s) [noise=true , address=77.90.31.19:4842 , piece_id=F3HBHRAXI5NGMDJG45SMQLSXTMOSYF6LKONDS2EGDQSPUYVKC46A, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.435918947 (408h59m38.307001982s) [noise=true , address=45.91.116.8:4046 , piece_id=UOUD2GKLRWTKQZRAGFYNFVIPHZROYUEDHTONEKGMPGIPC76OSF5A, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.469142901 (408h59m38.340225946s) [noise=true , address=45.91.116.10:4081 , piece_id=CQSKYN6XKXE3ASKW7GH6VRJWBYCM2XWN7OBCNCTVQ4HKDH3ZRYJQ, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.480038468 (408h59m38.351121503s) [noise=true , address=45.91.116.6:4975 , piece_id=XXE77DC5TGSPEZIDDEPNB4UHYI7JGVSRNCPMQI7O4YBVWL6AKCOA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.495720900 (408h59m38.366803945s) [noise=true , address=77.90.31.11:4556 , piece_id=YFOSG6X3FUKPAQDNUS46QEN6TKPVZU7T23DU26HEJAHGJLTQFI4Q, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.520490235 (408h59m38.39157327s) [noise=true , address=77.90.31.16:4736 , piece_id=Z5EJU25C6OEOAC3C2ZMCZ73FLPU6JRGFFMYHMGNCEO6G7KSJXHUA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:38.731918258 (408h59m38.603001293s) [noise=true , address=45.91.116.9:4262 , piece_id=LXZLGW2OB7ZTETME3ICJS5ZJB66NIWBDBZ46EXL3RMDENNUPKVKA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.059878596 (408h59m38.930961631s) [noise=true , address=145.79.227.5:4221 , piece_id=APBDNO5HES7F6TIMCA3WK4IUFJULOELJBOV2HBJH7RMVCJRRHFZA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.120519185 (408h59m38.99160224s) [noise=true , address=145.79.227.52:4447 , piece_id=73A5JY7AKKUW3GFH4FTYIHGTCMMBIHU5F3NLOJVBOJMIP2SB6PSQ, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.130571977 (408h59m39.001655012s) [noise=true , address=145.79.227.9:4366 , piece_id=FQVPTBM2QXP75XDUNPB534QUXZMUM5EJHME73GV4G2X6YQJ2SWPA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.144319064 (408h59m39.015402099s) [noise=true , address=145.79.227.38:4977 , piece_id=UY4TXHP6MVSX5L77MYZDXUE34OAZHLCEDQFO6SGQXIN2XEONFY4A, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.192087993 (408h59m39.063171028s) [noise=true , address=145.79.227.10:4401 , piece_id=CB72VDTVIGXKML6S6J5XP5RRRUP6TKYYQBUFDPSTUCMSCUCF2XIQ, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.247130523 (408h59m39.118213558s) [noise=true , address=145.79.227.39:4015 , piece_id=FYLJN3QH5SC5NIR4M7BZMWF4WMQXLKSCRFQV54GHKL573QZ7FLXA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: <nil> Dec 22 16:28:39.247151553 (408h59m39.118234588s) [seg_pos={0 0}, upload=359051]: Segment reached optimal threshold of 13 pieces. Dec 22 16:28:39.247340253 (408h59m39.118423288s) [noise=true , address=45.91.116.11:4120 , piece_id=IJ4DEJNJ2GURQF3T5XA6ZXXQLNPRH7QBMWCW54JFWWJBKFIIZTBA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 22 16:28:39.247347166 (408h59m39.118430201s) [noise=true , address=45.91.116.11:4120 , piece_id=IJ4DEJNJ2GURQF3T5XA6ZXXQLNPRH7QBMWCW54JFWWJBKFIIZTBA, seg_pos={0 0}, upload=359051]: Long tail cancellation due to optimal threshold Dec 22 16:28:39.247744692 (408h59m39.118827737s) [noise=true , address=145.79.227.20:4438 , piece_id=Z7Z3GJIH5U7JM5NFXW24KVDXNEVOWQZ3GY6423OPGC3DATWV6GCA, seg_pos={0 0}, upload=359051]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT): protocol: expected piece hash; context canceled; EOF Dec 22 16:28:39.247781953 (408h59m39.118865008s) [noise=true , address=145.79.227.20:4438 , piece_id=Z7Z3GJIH5U7JM5NFXW24KVDXNEVOWQZ3GY6423OPGC3DATWV6GCA, seg_pos={0 0}, upload=359051]: Long tail cancellation due to optimal threshold Dec 22 16:28:39.247810608 (408h59m39.118893653s) [seg_pos={0 0}, upload=359051]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:28:39.247971094 (408h59m39.119054139s) [upload=359051]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:28:39.247978087 (408h59m39.119061122s) [upload=359051]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:28:39.671766327 (408h59m39.542849372s) [upload=359051]: Done uploading segments... Dec 22 16:28:39.671799079 (408h59m39.542882124s) [upload=359051]: Upload finished. err: <nil> Duration: 30.003535375s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.447119284s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:29:00 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.447142477s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 466.705531ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:28:55 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 466.731721ms core satellite eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..XXXX....X..X...XX.....XXXX......X.X.X.X...X....XXXXXXX..X] duration: 30.001879229s trace: https://jaeger.dev.storj.io/trace/199acf39d5005139 FAIL (2025-12-22 16:28:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 22 16:28:05.287560304 (408h59m5.158643349s) [upload=359044]: Starting upload Dec 22 16:28:05.287780723 (408h59m5.158863768s) [upload=359044]: Done starting upload Dec 22 16:28:05.287904289 (408h59m5.158987334s) [upload=359044]: Uploading segments... Dec 22 16:28:05.288696427 (408h59m5.159779462s) [upload=359044]: Got next segment. Inline: false Dec 22 16:28:05.288777612 (408h59m5.159860657s) [upload=359044]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 22 16:28:05.288785346 (408h59m5.159868391s) [upload=359044]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 22 16:28:05.532320004 (408h59m5.40340305s) [seg_pos={0 0}, upload=359044]: Begin upload segment... Dec 22 16:28:05.532393174 (408h59m5.403476209s) [seg_pos={0 0}, upload=359044]: Done begin upload segment. Dec 22 16:28:05.532448600 (408h59m5.403531645s) [noise=true , address=45.91.116.14:4191 , piece_id=RBJISVWTFCXIKTXXYVGNFDOGDG34PBOBP6MGIF4WDLGSZOCTOPAA, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.532514986 (408h59m5.403598031s) [noise=true , address=145.79.227.15:4875 , piece_id=OUGUWKHDVGA2YM7NXPAF3REYYYSHVTRMHDMQXNJRTIPQHOQGCTPA, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.533186855 (408h59m5.40426989s) [noise=true , address=145.79.227.37:4565 , piece_id=DHCLI24TQRF743LK4VFSUPJCWEHPD4DSVXZVT2X3RMLMFIQNPF7Q, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.533333604 (408h59m5.404416649s) [noise=true , address=145.79.227.35:4481 , piece_id=3XWRB54GIQM3TR4I375H26BNCXNPSTSDA3XMQK5BFTLYXVXQKHZQ, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.533654326 (408h59m5.404737371s) [noise=true , address=45.91.116.5:4299 , piece_id=L5WAIFTX5XYLYEUMZVESDUX7YI6UMREKCEQKO5EW3R5MF5MRI2ZA, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.533997179 (408h59m5.405080214s) [noise=true , address=145.79.227.34:4906 , piece_id=XSNGCKMTYNKYUV2NCXJM7M733MCGKWC4TKUWTQCA3PVAJFVIMATQ, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.534328980 (408h59m5.405412025s) [noise=true , address=145.79.227.52:4446 , piece_id=K2ITHKS4NUH2V5DSEYLEMV2ACSY5BKIRD2HUU5J7DGQ3K3XXQC7Q, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.534769319 (408h59m5.405852374s) [noise=true , address=145.79.227.50:4375 , piece_id=ICIDSXYJ4TZ7DUO2O5XDKAYR6UY72XTZ7BJZJGXFXKILCXAWVMHQ, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.534854601 (408h59m5.405937636s) [noise=true , address=77.90.31.15:4697 , piece_id=YUE2H2XSA3MDZ7WWSTZRHO4YFVGWIONYLWQHRTTR3B2CUXYKBGZA, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.535298717 (408h59m5.406381762s) [noise=true , address=145.79.227.47:4265 , piece_id=NPZEZJBDIRY5Y4NNAWX5A4IMV63SIJP54JTJASCAPAFFZ3356I3A, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.535517874 (408h59m5.406600909s) [noise=true , address=145.79.227.17:4972 , piece_id=F3QDINHIQMZLMWWAKPBSMHKTXCH7KN4BYLZMATKFCPHLDHD6H3VA, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.535758542 (408h59m5.406841577s) [noise=true , address=145.79.227.38:4978 , piece_id=KT3RH3DM33KQIOHEKYP43NUXX364VBPQGUKODN3HVG2CUTXVTF6A, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.536388872 (408h59m5.407471917s) [noise=true , address=145.79.227.42:4121 , piece_id=N7QWF7IKOCERBQMJZ6QL7KKRZV4AVM6FPLWUNAJ3M4LNRIKE5V7A, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.537290079 (408h59m5.408373144s) [noise=true , address=77.90.31.18:4808 , piece_id=FF5RMNYAWPIF4HGRA4C7KSFIH73GV2IMMPKB75UZYYC7T4NUDBKQ, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.537908827 (408h59m5.408991872s) [noise=true , address=145.79.227.39:4015 , piece_id=53I7BK2XNQC4Q6EPIG7AEGWFMQIO54HS2NCXTBFEQM4Y2AJKHGHQ, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:05.538285685 (408h59m5.40936873s) [noise=true , address=77.90.31.10:4518 , piece_id=F23FJVE3AOQRK3VZOPPPKO42NWUG2J3KU4VMQHSKRU2FHTN3Q2HQ, seg_pos={0 0}, upload=359044]: Uploading piece... Dec 22 16:28:07.290146616 (408h59m7.161229661s) [upload=359044]: Next returned nil segment Dec 22 16:28:07.290165702 (408h59m7.161248747s) [upload=359044]: Waiting for error group managing segments... Dec 22 16:28:07.744998236 (408h59m7.616081281s) [noise=true , address=77.90.31.18:4808 , piece_id=FF5RMNYAWPIF4HGRA4C7KSFIH73GV2IMMPKB75UZYYC7T4NUDBKQ, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:07.879942151 (408h59m7.751025186s) [noise=true , address=45.91.116.14:4191 , piece_id=RBJISVWTFCXIKTXXYVGNFDOGDG34PBOBP6MGIF4WDLGSZOCTOPAA, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:07.888092529 (408h59m7.759175574s) [noise=true , address=77.90.31.15:4697 , piece_id=YUE2H2XSA3MDZ7WWSTZRHO4YFVGWIONYLWQHRTTR3B2CUXYKBGZA, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:07.952004177 (408h59m7.823087222s) [noise=true , address=45.91.116.5:4299 , piece_id=L5WAIFTX5XYLYEUMZVESDUX7YI6UMREKCEQKO5EW3R5MF5MRI2ZA, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:08.023080738 (408h59m7.894163774s) [noise=true , address=77.90.31.10:4518 , piece_id=F23FJVE3AOQRK3VZOPPPKO42NWUG2J3KU4VMQHSKRU2FHTN3Q2HQ, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:08.474210682 (408h59m8.345293717s) [noise=true , address=145.79.227.52:4446 , piece_id=K2ITHKS4NUH2V5DSEYLEMV2ACSY5BKIRD2HUU5J7DGQ3K3XXQC7Q, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:08.478929167 (408h59m8.350012202s) [noise=true , address=145.79.227.39:4015 , piece_id=53I7BK2XNQC4Q6EPIG7AEGWFMQIO54HS2NCXTBFEQM4Y2AJKHGHQ, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:08.483834477 (408h59m8.354917512s) [noise=true , address=145.79.227.17:4972 , piece_id=F3QDINHIQMZLMWWAKPBSMHKTXCH7KN4BYLZMATKFCPHLDHD6H3VA, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:08.487990901 (408h59m8.359073936s) [noise=true , address=145.79.227.38:4978 , piece_id=KT3RH3DM33KQIOHEKYP43NUXX364VBPQGUKODN3HVG2CUTXVTF6A, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:08.499915819 (408h59m8.370998854s) [noise=true , address=145.79.227.50:4375 , piece_id=ICIDSXYJ4TZ7DUO2O5XDKAYR6UY72XTZ7BJZJGXFXKILCXAWVMHQ, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:08.520581760 (408h59m8.391664795s) [noise=true , address=145.79.227.15:4875 , piece_id=OUGUWKHDVGA2YM7NXPAF3REYYYSHVTRMHDMQXNJRTIPQHOQGCTPA, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:08.533745856 (408h59m8.404828891s) [noise=true , address=145.79.227.42:4121 , piece_id=N7QWF7IKOCERBQMJZ6QL7KKRZV4AVM6FPLWUNAJ3M4LNRIKE5V7A, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:08.537761532 (408h59m8.408844578s) [noise=true , address=145.79.227.47:4265 , piece_id=NPZEZJBDIRY5Y4NNAWX5A4IMV63SIJP54JTJASCAPAFFZ3356I3A, seg_pos={0 0}, upload=359044]: Done uploading piece. err: <nil> Dec 22 16:28:08.537783184 (408h59m8.408866229s) [seg_pos={0 0}, upload=359044]: Segment reached optimal threshold of 13 pieces. Dec 22 16:28:08.537998905 (408h59m8.40908195s) [noise=true , address=145.79.227.35:4481 , piece_id=3XWRB54GIQM3TR4I375H26BNCXNPSTSDA3XMQK5BFTLYXVXQKHZQ, seg_pos={0 0}, upload=359044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 22 16:28:08.538004996 (408h59m8.409088031s) [noise=true , address=145.79.227.35:4481 , piece_id=3XWRB54GIQM3TR4I375H26BNCXNPSTSDA3XMQK5BFTLYXVXQKHZQ, seg_pos={0 0}, upload=359044]: Long tail cancellation due to optimal threshold Dec 22 16:28:08.538256174 (408h59m8.409339219s) [noise=true , address=145.79.227.34:4906 , piece_id=XSNGCKMTYNKYUV2NCXJM7M733MCGKWC4TKUWTQCA3PVAJFVIMATQ, seg_pos={0 0}, upload=359044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 22 16:28:08.538262848 (408h59m8.409345893s) [noise=true , address=145.79.227.34:4906 , piece_id=XSNGCKMTYNKYUV2NCXJM7M733MCGKWC4TKUWTQCA3PVAJFVIMATQ, seg_pos={0 0}, upload=359044]: Long tail cancellation due to optimal threshold Dec 22 16:28:08.538343932 (408h59m8.409426987s) [noise=true , address=145.79.227.37:4565 , piece_id=DHCLI24TQRF743LK4VFSUPJCWEHPD4DSVXZVT2X3RMLMFIQNPF7Q, seg_pos={0 0}, upload=359044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 22 16:28:08.538365263 (408h59m8.409448308s) [noise=true , address=145.79.227.37:4565 , piece_id=DHCLI24TQRF743LK4VFSUPJCWEHPD4DSVXZVT2X3RMLMFIQNPF7Q, seg_pos={0 0}, upload=359044]: Long tail cancellation due to optimal threshold Dec 22 16:28:08.538408505 (408h59m8.40949155s) [seg_pos={0 0}, upload=359044]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 22 16:28:08.538525918 (408h59m8.409608963s) [upload=359044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 22 16:28:08.538535185 (408h59m8.40961823s) [upload=359044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 22 16:28:08.832363401 (408h59m8.703446446s) [upload=359044]: Done uploading segments... Dec 22 16:28:08.832384191 (408h59m8.703467236s) [upload=359044]: Upload finished. err: <nil> Duration: 30.001904336s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X............................X...........................] duration: 5.337910946s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:28:14 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.337940474s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.414134666s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:28:14 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.414163259s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 471.055494ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:28:10 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 471.116461ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.396673757s trace: https://jaeger.dev.storj.io/trace/6ea3fd74aff47917 FAIL (2025-12-22 16:27:57 UTC) orbiter warning: operation: StatBucket uplink: bucket: metaclient: context deadline exceeded Duration: 1.396701631s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 462.399883ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:27:46 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 462.423397ms core satellite eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X............................X..........................] duration: 5.833310066s trace: https://jaeger.dev.storj.io/trace/72726d8466c916ea FAIL (2025-12-22 16:27:39 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 5.833337928s core uplink eu1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 473.962098ms target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/704b2d4a066e7708 FAIL (2025-12-22 16:27:11 UTC) orbiter warning: Get "https://eu1.storj.io/login": dial tcp 35.207.121.91:443: i/o timeout Duration: 473.992567ms core satellite eu1

Orbiter Check History

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)