Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 843066 reports (Page 11987 of 16862)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 34.19831366s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:16:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 34.198443296s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.436946788s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:15:54 UTC) orbiter warning: context deadline exceeded Duration: 3.436976907s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [X..............] duration: 316.487275ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:15:30 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 316.799049ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 45.71382256s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:15:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 45.713862537s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XX.X] duration: 30.257887063s trace: https://jaeger.dev.storj.io/trace/2580d0b7c20c3f52 FAIL (2025-11-11 05:15:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== eg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.038898143 (45h10m29.590368604s) [noise=true , address=77.90.31.18:4805 , piece_id=2QZLMROCMTYMGXHCSU4L3BM6YHUB4HYDVHFB3YK2IEYQFWFSCMYA, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.039323634 (45h10m29.590794095s) [noise=true , address=77.90.31.13:4627 , piece_id=UABE74GGGS6CE6UDC6JFSL753WG5G5RAGJRFDZ432SOHV5A2BE3A, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.039737231 (45h10m29.591207693s) [noise=true , address=45.91.116.13:4155 , piece_id=DCFLKTKYF5X46PVYOQ7P4RGFBZTB6MQ5Y56GCOLWTCGUEF2MCHFA, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.040160557 (45h10m29.591631018s) [noise=true , address=45.91.116.4:4937 , piece_id=A7PBGLUN4PBL22Q5VHHIJVYW3PDL7LNVJ6DC5I5MPQRLNOSX5NNA, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.040746573 (45h10m29.592217035s) [noise=true , address=145.79.227.22:4510 , piece_id=TU6MEAXRSTEHTY4LMDAG2PA5RTG52LYVDTBWARUHQKXSXLTG33NA, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.041170200 (45h10m29.592640661s) [noise=true , address=77.90.31.11:4554 , piece_id=N56Y3Z4FCCTOL6IQPEUBKE5HSDBGZXU2A7ZCOSNCI7QA5EZZLLEQ, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.041640204 (45h10m29.593110655s) [noise=true , address=145.79.227.20:4439 , piece_id=MWKNABR3MHBJZBPEDKODBKSRX3XFQPTOAFDSUXIDOJ5Z65RIFXAA, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.042220269 (45h10m29.59369074s) [noise=true , address=145.79.227.21:4473 , piece_id=6DEVG6NQSY5X54SNF544U62WYIGKYFPK7YLCDVSDD2HD325YCRTQ, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.043867916 (45h10m29.595338377s) [noise=true , address=145.79.227.26:4655 , piece_id=QRCPKUQ6O6B6HIZOXGH2FJHXRCH4SMNOY2XAWZ57S2FRVIZGAP2Q, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.046841469 (45h10m29.59831192s) [noise=true , address=45.91.116.11:4118 , piece_id=DXJWO2OJTXTPOQMFOIU52FNHRXWTSOP7S2B45ZFN2MVMTC3SGPMA, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.055249099 (45h10m29.60671956s) [noise=true , address=145.79.227.23:4545 , piece_id=Z5LRQV62BOTZPFE2Y7XTIS3ZY5SCDVOB3DU4RK4QLCO4NSWCG6MQ, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.058103865 (45h10m29.609574326s) [noise=true , address=145.79.227.8:4331 , piece_id=FLZADZ6CPG6FLDX7EOSIAJMMH7FEAXSKL5EDOBZVNCI3ERV7STTQ, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.062579256 (45h10m29.614049718s) [noise=true , address=145.79.227.16:4909 , piece_id=T7RMZV7PKDHQO7SYIU4N7TLADIGIPXBYAYRHITKCEE2PE7FKVX6Q, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:31.064389293 (45h10m29.615859754s) [noise=true , address=45.91.116.8:4045 , piece_id=TLXW4D66T7AUHNMKIC3PSPV4JR7OIAJW4NXIVFCWQFCDSYEHYU2A, seg_pos={0 0}, upload=39815]: Uploading piece... Nov 11 05:14:32.797364784 (45h10m31.348835245s) [upload=39815]: Next returned nil segment Nov 11 05:14:32.797382036 (45h10m31.348852497s) [upload=39815]: Waiting for error group managing segments... Nov 11 05:14:33.119691668 (45h10m31.671162119s) [noise=true , address=77.90.31.17:4772 , piece_id=HWQVVWGAPLI7BNOQFKNTHBLGNLENKHYQ77VI52246BKS3PPT3KDQ, seg_pos={0 0}, upload=39815]: Done uploading piece. err: <nil> Nov 11 05:14:33.145170235 (45h10m31.696640696s) [noise=true , address=77.90.31.13:4627 , piece_id=UABE74GGGS6CE6UDC6JFSL753WG5G5RAGJRFDZ432SOHV5A2BE3A, seg_pos={0 0}, upload=39815]: Done uploading piece. err: <nil> Nov 11 05:14:33.146703755 (45h10m31.698174216s) [noise=true , address=45.91.116.13:4155 , piece_id=DCFLKTKYF5X46PVYOQ7P4RGFBZTB6MQ5Y56GCOLWTCGUEF2MCHFA, seg_pos={0 0}, upload=39815]: Done uploading piece. err: <nil> Nov 11 05:14:33.174484765 (45h10m31.725955216s) [noise=true , address=77.90.31.11:4554 , piece_id=N56Y3Z4FCCTOL6IQPEUBKE5HSDBGZXU2A7ZCOSNCI7QA5EZZLLEQ, seg_pos={0 0}, upload=39815]: Done uploading piece. err: <nil> Nov 11 05:14:33.180422792 (45h10m31.731893243s) [noise=true , address=45.91.116.11:4118 , piece_id=DXJWO2OJTXTPOQMFOIU52FNHRXWTSOP7S2B45ZFN2MVMTC3SGPMA, seg_pos={0 0}, upload=39815]: Done uploading piece. err: <nil> Nov 11 05:14:33.210640002 (45h10m31.762110463s) [noise=true , address=77.90.31.18:4805 , piece_id=2QZLMROCMTYMGXHCSU4L3BM6YHUB4HYDVHFB3YK2IEYQFWFSCMYA, seg_pos={0 0}, upload=39815]: Done uploading piece. err: <nil> Nov 11 05:14:33.237793618 (45h10m31.789264079s) [noise=true , address=45.91.116.8:4045 , piece_id=TLXW4D66T7AUHNMKIC3PSPV4JR7OIAJW4NXIVFCWQFCDSYEHYU2A, seg_pos={0 0}, upload=39815]: Done uploading piece. err: <nil> Nov 11 05:14:33.242251657 (45h10m31.793722108s) [noise=true , address=45.91.116.4:4937 , piece_id=A7PBGLUN4PBL22Q5VHHIJVYW3PDL7LNVJ6DC5I5MPQRLNOSX5NNA, seg_pos={0 0}, upload=39815]: Done uploading piece. err: <nil> Nov 11 05:15:00.082327234 (45h10m58.633797695s) [noise=true , address=145.79.227.23:4545 , piece_id=Z5LRQV62BOTZPFE2Y7XTIS3ZY5SCDVOB3DU4RK4QLCO4NSWCG6MQ, seg_pos={0 0}, upload=39815]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:36298->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:36298->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36298->145.79.227.23:4545: use of closed network connection Nov 11 05:15:00.082496327 (45h10m58.633966788s) [noise=true , address=145.79.227.22:4510 , piece_id=TU6MEAXRSTEHTY4LMDAG2PA5RTG52LYVDTBWARUHQKXSXLTG33NA, seg_pos={0 0}, upload=39815]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: write tcp 188.166.225.170:56762->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:56762->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56762->145.79.227.22:4510: use of closed network connection Nov 11 05:15:00.082676199 (45h10m58.63414665s) [noise=true , address=145.79.227.26:4655 , piece_id=QRCPKUQ6O6B6HIZOXGH2FJHXRCH4SMNOY2XAWZ57S2FRVIZGAP2Q, seg_pos={0 0}, upload=39815]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:58746->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:58746->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58746->145.79.227.26:4655: use of closed network connection Nov 11 05:15:00.082707458 (45h10m58.63417793s) [noise=true , address=145.79.227.20:4439 , piece_id=MWKNABR3MHBJZBPEDKODBKSRX3XFQPTOAFDSUXIDOJ5Z65RIFXAA, seg_pos={0 0}, upload=39815]: Done uploading piece. err: ecclient: upload failed (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn, address:145.79.227.20:4439): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:15:00.082832186 (45h10m58.634302647s) [noise=true , address=145.79.227.21:4473 , piece_id=6DEVG6NQSY5X54SNF544U62WYIGKYFPK7YLCDVSDD2HD325YCRTQ, seg_pos={0 0}, upload=39815]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:39840->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:39840->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39840->145.79.227.21:4473: use of closed network connection Nov 11 05:15:00.082884496 (45h10m58.634354967s) [noise=true , address=145.79.227.16:4909 , piece_id=T7RMZV7PKDHQO7SYIU4N7TLADIGIPXBYAYRHITKCEE2PE7FKVX6Q, seg_pos={0 0}, upload=39815]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:52452->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:52452->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52452->145.79.227.16:4909: use of closed network connection Nov 11 05:15:00.082977343 (45h10m58.634447804s) [noise=true , address=145.79.227.15:4875 , piece_id=QVOTKAKOXNIWSQR2EY4LJSDW5BG7GYRTEZ6WJBGJVAQWRPWM3OHA, seg_pos={0 0}, upload=39815]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: write tcp 188.166.225.170:39586->145.79.227.15:4875: use of closed network connection; write tcp 188.166.225.170:39586->145.79.227.15:4875: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39586->145.79.227.15:4875: use of closed network connection Nov 11 05:15:00.083077574 (45h10m58.634548035s) [noise=true , address=145.79.227.8:4331 , piece_id=FLZADZ6CPG6FLDX7EOSIAJMMH7FEAXSKL5EDOBZVNCI3ERV7STTQ, seg_pos={0 0}, upload=39815]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:37938->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:37938->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37938->145.79.227.8:4331: use of closed network connection Nov 11 05:15:00.083112169 (45h10m58.634582631s) [seg_pos={0 0}, upload=39815]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Nov 11 05:15:00.332226773 (45h10m58.883697224s) [upload=39815]: Done uploading segments... Nov 11 05:15:00.332443015 (45h10m58.883913476s) [upload=39815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.257914456s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.X.............] duration: 363.188675ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:15:00 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 363.217819ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.434199704s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:14:39 UTC) orbiter warning: context deadline exceeded Duration: 3.434238971s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..X............] duration: 344.377473ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:14:18 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 344.392171ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 5.918638759s trace: https://jaeger.dev.storj.io/trace/70d8a0f235382a72 FAIL (2025-11-11 05:13:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== (45h8m49.504299775s) [seg_pos={0 0}, upload=39791]: Done begin upload segment. Nov 11 05:12:50.952849041 (45h8m49.504319502s) [noise=true , address=45.91.116.8:4045 , piece_id=RJ6GWL6ILYIPCK6TT6D3EMBPQ4EFKWCOFNRNSHAFMD7AXCBVMXRQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.952868638 (45h8m49.504339089s) [noise=true , address=145.79.227.26:4654 , piece_id=5C5ZHIGMQBR2YCROMG42OJLNHGND3CEOX73QDX6SOCR7HS7SXRXQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.953451368 (45h8m49.504921829s) [noise=true , address=145.79.227.15:4874 , piece_id=SRGS5CQLAQACA36IMJQYMZNLPAOQAAPMM3LYWMXUPFULZNNOFGDA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.953967601 (45h8m49.505438062s) [noise=true , address=77.90.31.14:4661 , piece_id=6OZK3ZW37UW5BDCJHCAJDBAJJ4O2GP4OCBMJGGXLBHS2TGX2H3KQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.954266951 (45h8m49.505737412s) [noise=true , address=145.79.227.16:4911 , piece_id=3YHLQIM2QWJL376MJLQCPVH6XQMBT5FJGVVUMXVRQFVFO4U2O6IQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.954571792 (45h8m49.506042253s) [noise=true , address=145.79.227.24:4583 , piece_id=F7IQRHXCRALUDREKCUJ6OTND4MH6Y5BWFVXDROYISFVATGHS3LVQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.954961624 (45h8m49.506432085s) [noise=true , address=145.79.227.32:4836 , piece_id=JMSBKBYJBDGL62KR4VBRSKLNPBCWW2JZ5YSWISMTOBLI67WKFMWA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.955379529 (45h8m49.50684999s) [noise=true , address=145.79.227.9:4365 , piece_id=P5OPRZ6CIAFFDAHASYKMVA6XIPRJHQDFLFCHWMQASWJHHUT6RISQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.955728875 (45h8m49.507199336s) [noise=true , address=145.79.227.21:4474 , piece_id=R33VC7NOMZFH2XVR24R65LXMNQPUZHARZP66UOCSCNXSWQRZR3SA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.956105141 (45h8m49.507575602s) [noise=true , address=45.91.116.13:4153 , piece_id=USEZ5YDYK4LL5GF5FROXBDLZVOP5CRDUE2GC7OLF2AAKI56O43AA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.956506285 (45h8m49.507976746s) [noise=true , address=145.79.227.18:4006 , piece_id=6FL56ZEGKX3EPTEJNAOHW463SPAK6RPDOYEDMG3SMJHTG55DQVYA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.956961011 (45h8m49.508431472s) [noise=true , address=145.79.227.14:4840 , piece_id=WZDWYEKHG27EQQFTLVB3JA5NC6GICY3MD7J7BZI4S25ZGH4UTYHA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.958201543 (45h8m49.509671994s) [noise=true , address=145.79.227.10:4403 , piece_id=D77MVJNWCJZG7F3OEY53BPFRVZLDOH6OJBWTS5SDKSGQTMSEXNKQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.959489275 (45h8m49.510959736s) [noise=true , address=145.79.227.27:4691 , piece_id=U6W5TMUHGVG3WQHB4DIHYCRMR7YCBMESGFUVPQAXNNFSKQTY54QA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.961089223 (45h8m49.512559714s) [noise=true , address=145.79.227.10:4402 , piece_id=TATGO55KPU5K5Q5U6ZJCO4VJONBWA36REJUT2EDTM4TH2LTFNT6Q, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.962539103 (45h8m49.514009564s) [noise=true , address=145.79.227.8:4331 , piece_id=3DUEQVK3BJIHTOVDNXAGZRCXX2MJQURQXHPNWKYU3PHHUJKGPM3Q, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:52.687765458 (45h8m51.239235929s) [upload=39791]: Next returned nil segment Nov 11 05:12:52.687786980 (45h8m51.239257441s) [upload=39791]: Waiting for error group managing segments... Nov 11 05:12:53.026234221 (45h8m51.577704682s) [noise=true , address=45.91.116.13:4153 , piece_id=USEZ5YDYK4LL5GF5FROXBDLZVOP5CRDUE2GC7OLF2AAKI56O43AA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:12:53.032731143 (45h8m51.584201604s) [noise=true , address=45.91.116.8:4045 , piece_id=RJ6GWL6ILYIPCK6TT6D3EMBPQ4EFKWCOFNRNSHAFMD7AXCBVMXRQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:12:54.589913708 (45h8m53.141384169s) [noise=true , address=77.90.31.14:4661 , piece_id=6OZK3ZW37UW5BDCJHCAJDBAJJ4O2GP4OCBMJGGXLBHS2TGX2H3KQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:16.596399117 (45h9m15.147869578s) [noise=true , address=145.79.227.16:4911 , piece_id=3YHLQIM2QWJL376MJLQCPVH6XQMBT5FJGVVUMXVRQFVFO4U2O6IQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:17.134331208 (45h9m15.685801669s) [noise=true , address=145.79.227.18:4006 , piece_id=6FL56ZEGKX3EPTEJNAOHW463SPAK6RPDOYEDMG3SMJHTG55DQVYA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:17.491581624 (45h9m16.043052085s) [noise=true , address=145.79.227.27:4691 , piece_id=U6W5TMUHGVG3WQHB4DIHYCRMR7YCBMESGFUVPQAXNNFSKQTY54QA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:17.731997053 (45h9m16.283467514s) [noise=true , address=145.79.227.24:4583 , piece_id=F7IQRHXCRALUDREKCUJ6OTND4MH6Y5BWFVXDROYISFVATGHS3LVQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:18.466299551 (45h9m17.017770012s) [noise=true , address=145.79.227.32:4836 , piece_id=JMSBKBYJBDGL62KR4VBRSKLNPBCWW2JZ5YSWISMTOBLI67WKFMWA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:19.859666269 (45h9m18.41113674s) [noise=true , address=145.79.227.10:4403 , piece_id=D77MVJNWCJZG7F3OEY53BPFRVZLDOH6OJBWTS5SDKSGQTMSEXNKQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:19.991802834 (45h9m18.543273305s) [noise=true , address=145.79.227.21:4474 , piece_id=R33VC7NOMZFH2XVR24R65LXMNQPUZHARZP66UOCSCNXSWQRZR3SA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:40348->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:40348->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40348->145.79.227.21:4474: use of closed network connection Nov 11 05:13:19.992020498 (45h9m18.543490969s) [noise=true , address=145.79.227.15:4874 , piece_id=SRGS5CQLAQACA36IMJQYMZNLPAOQAAPMM3LYWMXUPFULZNNOFGDA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38246->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.15:4874: use of closed network connection Nov 11 05:13:19.992123595 (45h9m18.543594066s) [noise=true , address=145.79.227.9:4365 , piece_id=P5OPRZ6CIAFFDAHASYKMVA6XIPRJHQDFLFCHWMQASWJHHUT6RISQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:13:19.992150306 (45h9m18.543620767s) [noise=true , address=145.79.227.14:4840 , piece_id=WZDWYEKHG27EQQFTLVB3JA5NC6GICY3MD7J7BZI4S25ZGH4UTYHA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58618->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58618->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58618->145.79.227.14:4840: use of closed network connection Nov 11 05:13:19.992453774 (45h9m18.543924255s) [noise=true , address=145.79.227.10:4402 , piece_id=TATGO55KPU5K5Q5U6ZJCO4VJONBWA36REJUT2EDTM4TH2LTFNT6Q, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:60698->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:60698->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60698->145.79.227.10:4402: use of closed network connection Nov 11 05:13:19.992508678 (45h9m18.543979139s) [noise=true , address=145.79.227.26:4654 , piece_id=5C5ZHIGMQBR2YCROMG42OJLNHGND3CEOX73QDX6SOCR7HS7SXRXQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:58334->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:58334->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58334->145.79.227.26:4654: use of closed network connection Nov 11 05:13:19.992616643 (45h9m18.544087114s) [noise=true , address=145.79.227.8:4331 , piece_id=3DUEQVK3BJIHTOVDNXAGZRCXX2MJQURQXHPNWKYU3PHHUJKGPM3Q, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:56490->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:56490->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56490->145.79.227.8:4331: use of closed network connection Nov 11 05:13:19.992655779 (45h9m18.54412624s) [seg_pos={0 0}, upload=39791]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 11 05:13:20.241081020 (45h9m18.792551481s) [upload=39791]: Done uploading segments... Nov 11 05:13:20.241128981 (45h9m18.792599442s) [upload=39791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.918668084s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...X...........] duration: 342.989159ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:13:45 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 343.015589ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.430653139s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:13:33 UTC) orbiter warning: context deadline exceeded Duration: 3.430725416s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.732383041s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:13:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.732413538s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.251338475s trace: https://jaeger.dev.storj.io/trace/70d8a0f235382a72 FAIL (2025-11-11 05:13:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== (45h8m49.504299775s) [seg_pos={0 0}, upload=39791]: Done begin upload segment. Nov 11 05:12:50.952849041 (45h8m49.504319502s) [noise=true , address=45.91.116.8:4045 , piece_id=RJ6GWL6ILYIPCK6TT6D3EMBPQ4EFKWCOFNRNSHAFMD7AXCBVMXRQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.952868638 (45h8m49.504339089s) [noise=true , address=145.79.227.26:4654 , piece_id=5C5ZHIGMQBR2YCROMG42OJLNHGND3CEOX73QDX6SOCR7HS7SXRXQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.953451368 (45h8m49.504921829s) [noise=true , address=145.79.227.15:4874 , piece_id=SRGS5CQLAQACA36IMJQYMZNLPAOQAAPMM3LYWMXUPFULZNNOFGDA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.953967601 (45h8m49.505438062s) [noise=true , address=77.90.31.14:4661 , piece_id=6OZK3ZW37UW5BDCJHCAJDBAJJ4O2GP4OCBMJGGXLBHS2TGX2H3KQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.954266951 (45h8m49.505737412s) [noise=true , address=145.79.227.16:4911 , piece_id=3YHLQIM2QWJL376MJLQCPVH6XQMBT5FJGVVUMXVRQFVFO4U2O6IQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.954571792 (45h8m49.506042253s) [noise=true , address=145.79.227.24:4583 , piece_id=F7IQRHXCRALUDREKCUJ6OTND4MH6Y5BWFVXDROYISFVATGHS3LVQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.954961624 (45h8m49.506432085s) [noise=true , address=145.79.227.32:4836 , piece_id=JMSBKBYJBDGL62KR4VBRSKLNPBCWW2JZ5YSWISMTOBLI67WKFMWA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.955379529 (45h8m49.50684999s) [noise=true , address=145.79.227.9:4365 , piece_id=P5OPRZ6CIAFFDAHASYKMVA6XIPRJHQDFLFCHWMQASWJHHUT6RISQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.955728875 (45h8m49.507199336s) [noise=true , address=145.79.227.21:4474 , piece_id=R33VC7NOMZFH2XVR24R65LXMNQPUZHARZP66UOCSCNXSWQRZR3SA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.956105141 (45h8m49.507575602s) [noise=true , address=45.91.116.13:4153 , piece_id=USEZ5YDYK4LL5GF5FROXBDLZVOP5CRDUE2GC7OLF2AAKI56O43AA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.956506285 (45h8m49.507976746s) [noise=true , address=145.79.227.18:4006 , piece_id=6FL56ZEGKX3EPTEJNAOHW463SPAK6RPDOYEDMG3SMJHTG55DQVYA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.956961011 (45h8m49.508431472s) [noise=true , address=145.79.227.14:4840 , piece_id=WZDWYEKHG27EQQFTLVB3JA5NC6GICY3MD7J7BZI4S25ZGH4UTYHA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.958201543 (45h8m49.509671994s) [noise=true , address=145.79.227.10:4403 , piece_id=D77MVJNWCJZG7F3OEY53BPFRVZLDOH6OJBWTS5SDKSGQTMSEXNKQ, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.959489275 (45h8m49.510959736s) [noise=true , address=145.79.227.27:4691 , piece_id=U6W5TMUHGVG3WQHB4DIHYCRMR7YCBMESGFUVPQAXNNFSKQTY54QA, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.961089223 (45h8m49.512559714s) [noise=true , address=145.79.227.10:4402 , piece_id=TATGO55KPU5K5Q5U6ZJCO4VJONBWA36REJUT2EDTM4TH2LTFNT6Q, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:50.962539103 (45h8m49.514009564s) [noise=true , address=145.79.227.8:4331 , piece_id=3DUEQVK3BJIHTOVDNXAGZRCXX2MJQURQXHPNWKYU3PHHUJKGPM3Q, seg_pos={0 0}, upload=39791]: Uploading piece... Nov 11 05:12:52.687765458 (45h8m51.239235929s) [upload=39791]: Next returned nil segment Nov 11 05:12:52.687786980 (45h8m51.239257441s) [upload=39791]: Waiting for error group managing segments... Nov 11 05:12:53.026234221 (45h8m51.577704682s) [noise=true , address=45.91.116.13:4153 , piece_id=USEZ5YDYK4LL5GF5FROXBDLZVOP5CRDUE2GC7OLF2AAKI56O43AA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:12:53.032731143 (45h8m51.584201604s) [noise=true , address=45.91.116.8:4045 , piece_id=RJ6GWL6ILYIPCK6TT6D3EMBPQ4EFKWCOFNRNSHAFMD7AXCBVMXRQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:12:54.589913708 (45h8m53.141384169s) [noise=true , address=77.90.31.14:4661 , piece_id=6OZK3ZW37UW5BDCJHCAJDBAJJ4O2GP4OCBMJGGXLBHS2TGX2H3KQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:16.596399117 (45h9m15.147869578s) [noise=true , address=145.79.227.16:4911 , piece_id=3YHLQIM2QWJL376MJLQCPVH6XQMBT5FJGVVUMXVRQFVFO4U2O6IQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:17.134331208 (45h9m15.685801669s) [noise=true , address=145.79.227.18:4006 , piece_id=6FL56ZEGKX3EPTEJNAOHW463SPAK6RPDOYEDMG3SMJHTG55DQVYA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:17.491581624 (45h9m16.043052085s) [noise=true , address=145.79.227.27:4691 , piece_id=U6W5TMUHGVG3WQHB4DIHYCRMR7YCBMESGFUVPQAXNNFSKQTY54QA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:17.731997053 (45h9m16.283467514s) [noise=true , address=145.79.227.24:4583 , piece_id=F7IQRHXCRALUDREKCUJ6OTND4MH6Y5BWFVXDROYISFVATGHS3LVQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:18.466299551 (45h9m17.017770012s) [noise=true , address=145.79.227.32:4836 , piece_id=JMSBKBYJBDGL62KR4VBRSKLNPBCWW2JZ5YSWISMTOBLI67WKFMWA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:19.859666269 (45h9m18.41113674s) [noise=true , address=145.79.227.10:4403 , piece_id=D77MVJNWCJZG7F3OEY53BPFRVZLDOH6OJBWTS5SDKSGQTMSEXNKQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: <nil> Nov 11 05:13:19.991802834 (45h9m18.543273305s) [noise=true , address=145.79.227.21:4474 , piece_id=R33VC7NOMZFH2XVR24R65LXMNQPUZHARZP66UOCSCNXSWQRZR3SA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: write tcp 188.166.225.170:40348->145.79.227.21:4474: use of closed network connection; write tcp 188.166.225.170:40348->145.79.227.21:4474: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40348->145.79.227.21:4474: use of closed network connection Nov 11 05:13:19.992020498 (45h9m18.543490969s) [noise=true , address=145.79.227.15:4874 , piece_id=SRGS5CQLAQACA36IMJQYMZNLPAOQAAPMM3LYWMXUPFULZNNOFGDA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: write tcp 188.166.225.170:38246->145.79.227.15:4874: use of closed network connection; write tcp 188.166.225.170:38246->145.79.227.15:4874: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38246->145.79.227.15:4874: use of closed network connection Nov 11 05:13:19.992123595 (45h9m18.543594066s) [noise=true , address=145.79.227.9:4365 , piece_id=P5OPRZ6CIAFFDAHASYKMVA6XIPRJHQDFLFCHWMQASWJHHUT6RISQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:13:19.992150306 (45h9m18.543620767s) [noise=true , address=145.79.227.14:4840 , piece_id=WZDWYEKHG27EQQFTLVB3JA5NC6GICY3MD7J7BZI4S25ZGH4UTYHA, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:145.79.227.14:4840): protocol: write tcp 188.166.225.170:58618->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:58618->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58618->145.79.227.14:4840: use of closed network connection Nov 11 05:13:19.992453774 (45h9m18.543924255s) [noise=true , address=145.79.227.10:4402 , piece_id=TATGO55KPU5K5Q5U6ZJCO4VJONBWA36REJUT2EDTM4TH2LTFNT6Q, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:60698->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:60698->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60698->145.79.227.10:4402: use of closed network connection Nov 11 05:13:19.992508678 (45h9m18.543979139s) [noise=true , address=145.79.227.26:4654 , piece_id=5C5ZHIGMQBR2YCROMG42OJLNHGND3CEOX73QDX6SOCR7HS7SXRXQ, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:58334->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:58334->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58334->145.79.227.26:4654: use of closed network connection Nov 11 05:13:19.992616643 (45h9m18.544087114s) [noise=true , address=145.79.227.8:4331 , piece_id=3DUEQVK3BJIHTOVDNXAGZRCXX2MJQURQXHPNWKYU3PHHUJKGPM3Q, seg_pos={0 0}, upload=39791]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:56490->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:56490->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56490->145.79.227.8:4331: use of closed network connection Nov 11 05:13:19.992655779 (45h9m18.54412624s) [seg_pos={0 0}, upload=39791]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 11 05:13:20.241081020 (45h9m18.792551481s) [upload=39791]: Done uploading segments... Nov 11 05:13:20.241128981 (45h9m18.792599442s) [upload=39791]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.251362611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 32.42018293s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:13:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.420202537s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 328.368819ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:13:01 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 328.400379ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.27041589s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:12:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.270469993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.255512815s trace: https://jaeger.dev.storj.io/trace/4d09e135e3da7327 FAIL (2025-11-11 05:12:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== IZX6PZK5QCSJSPER54S3Q, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.692644267 (45h8m5.244114738s) [noise=true , address=77.90.31.17:4770 , piece_id=HEO4X52YY6DPKEAPGLR26CCXDMPKR2GF2EDERRM6YE7VOBLTMKBA, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.694113875 (45h8m5.245584336s) [noise=true , address=145.79.227.23:4546 , piece_id=3RVLJXG52C5NBKCVZZ3SRGGUI6VNXGLIMNKVBHXSEAKTXDYRX5ZQ, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.694615821 (45h8m5.246086282s) [noise=true , address=45.91.116.14:4189 , piece_id=4FWYWUQ5WHH63T74ACSNCFGRHH5OOOX5J2KEVLYSWSXHYUITHTJQ, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.695156350 (45h8m5.246626801s) [noise=true , address=145.79.227.31:4797 , piece_id=EY2RHNOF7GCHLERQQW7CCFEMVAZDEZ6KU3GYA4XVBBL26SBWQLPA, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.696338160 (45h8m5.247808621s) [noise=true , address=45.91.116.11:4118 , piece_id=K2HPICKGPBS6FN72GCNPZGNRXVGIP66T5YHIN4FUWIPAUOKV6KLQ, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.698954452 (45h8m5.250424913s) [noise=true , address=77.90.31.15:4698 , piece_id=XM4YLFB2XXWHFAX5NZOUW52AT4NDWLXOLGCPMFEJTDA6FHX7VX3Q, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.700447115 (45h8m5.251917576s) [noise=true , address=145.79.227.24:4581 , piece_id=5PUE4EYDSSDJKICNL6BWTDIYESTHBGCA2D55BPPGVCIUAAXCFU7Q, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.701299198 (45h8m5.252769679s) [noise=true , address=45.91.116.4:4937 , piece_id=XHFHW3XDBLCVIGVDXCTTHKOUNXDNL23QDYUVES2NNRSRTI5KI4SA, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.702498070 (45h8m5.253968531s) [noise=true , address=145.79.227.8:4331 , piece_id=A25M674WCTR4HXXH44AVWLU7NDJHJIHQYVZDVH6CKTP36ZLU3P4A, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.703053468 (45h8m5.254523929s) [noise=true , address=45.91.116.8:4048 , piece_id=AYYWQLTLLH3HHUJTBTHV7ACRMGE7UUUH4N7KSMXZPWDTXGUDLYMA, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.704531032 (45h8m5.256001483s) [noise=true , address=145.79.227.14:4839 , piece_id=ZX3EZM2KO53CECZJG3P7RVUNK5FGV2RHOXANOZXXWUF2SGYHY6OA, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.706480514 (45h8m5.257950975s) [noise=true , address=77.90.31.10:4520 , piece_id=TZWEKTCXXITSFMIEVKBSWQN3UPKTLRVVZW5CHK2HITNAAL3OTPKA, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.711316673 (45h8m5.262787144s) [noise=true , address=77.90.31.17:4772 , piece_id=GRUKYTQUY6GXCKDSDOI6JBI3ELT2763CT7EQEV73RPGXS2SR6EDA, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.714278713 (45h8m5.265749174s) [noise=true , address=145.79.227.27:4689 , piece_id=AYSB5RC47IOX5O75VXTXMRFMMNKFTFXVACAX5E7PEGFOWUJEHRWA, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:06.714811517 (45h8m5.266281978s) [noise=true , address=77.90.31.12:4592 , piece_id=O5V3UTDDIIOM7JRIG2K4ZLFII4ZIT4C42VA2ZQOXNNN7Z24O36FA, seg_pos={0 0}, upload=39782]: Uploading piece... Nov 11 05:12:08.383147186 (45h8m6.934617647s) [upload=39782]: Next returned nil segment Nov 11 05:12:08.383195498 (45h8m6.934665959s) [upload=39782]: Waiting for error group managing segments... Nov 11 05:12:08.788818655 (45h8m7.340289126s) [noise=true , address=45.91.116.14:4189 , piece_id=4FWYWUQ5WHH63T74ACSNCFGRHH5OOOX5J2KEVLYSWSXHYUITHTJQ, seg_pos={0 0}, upload=39782]: Done uploading piece. err: <nil> Nov 11 05:12:08.814871284 (45h8m7.366341746s) [noise=true , address=45.91.116.11:4118 , piece_id=K2HPICKGPBS6FN72GCNPZGNRXVGIP66T5YHIN4FUWIPAUOKV6KLQ, seg_pos={0 0}, upload=39782]: Done uploading piece. err: <nil> Nov 11 05:12:08.827740019 (45h8m7.37921048s) [noise=true , address=77.90.31.10:4520 , piece_id=TZWEKTCXXITSFMIEVKBSWQN3UPKTLRVVZW5CHK2HITNAAL3OTPKA, seg_pos={0 0}, upload=39782]: Done uploading piece. err: <nil> Nov 11 05:12:08.837534700 (45h8m7.389005162s) [noise=true , address=45.91.116.4:4937 , piece_id=XHFHW3XDBLCVIGVDXCTTHKOUNXDNL23QDYUVES2NNRSRTI5KI4SA, seg_pos={0 0}, upload=39782]: Done uploading piece. err: <nil> Nov 11 05:12:08.867848253 (45h8m7.419318704s) [noise=true , address=77.90.31.17:4772 , piece_id=GRUKYTQUY6GXCKDSDOI6JBI3ELT2763CT7EQEV73RPGXS2SR6EDA, seg_pos={0 0}, upload=39782]: Done uploading piece. err: <nil> Nov 11 05:12:08.872554214 (45h8m7.424024675s) [noise=true , address=45.91.116.8:4048 , piece_id=AYYWQLTLLH3HHUJTBTHV7ACRMGE7UUUH4N7KSMXZPWDTXGUDLYMA, seg_pos={0 0}, upload=39782]: Done uploading piece. err: <nil> Nov 11 05:12:08.890574619 (45h8m7.44204509s) [noise=true , address=77.90.31.17:4770 , piece_id=HEO4X52YY6DPKEAPGLR26CCXDMPKR2GF2EDERRM6YE7VOBLTMKBA, seg_pos={0 0}, upload=39782]: Done uploading piece. err: <nil> Nov 11 05:12:08.905668811 (45h8m7.457139262s) [noise=true , address=77.90.31.15:4698 , piece_id=XM4YLFB2XXWHFAX5NZOUW52AT4NDWLXOLGCPMFEJTDA6FHX7VX3Q, seg_pos={0 0}, upload=39782]: Done uploading piece. err: <nil> Nov 11 05:12:08.913106434 (45h8m7.464576895s) [noise=true , address=77.90.31.12:4592 , piece_id=O5V3UTDDIIOM7JRIG2K4ZLFII4ZIT4C42VA2ZQOXNNN7Z24O36FA, seg_pos={0 0}, upload=39782]: Done uploading piece. err: <nil> Nov 11 05:12:35.729751494 (45h8m34.281221965s) [noise=true , address=145.79.227.14:4839 , piece_id=ZX3EZM2KO53CECZJG3P7RVUNK5FGV2RHOXANOZXXWUF2SGYHY6OA, seg_pos={0 0}, upload=39782]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:41356->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:41356->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41356->145.79.227.14:4839: use of closed network connection Nov 11 05:12:35.729962706 (45h8m34.281433167s) [noise=true , address=145.79.227.23:4546 , piece_id=3RVLJXG52C5NBKCVZZ3SRGGUI6VNXGLIMNKVBHXSEAKTXDYRX5ZQ, seg_pos={0 0}, upload=39782]: Done uploading piece. err: ecclient: upload failed (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5, address:145.79.227.23:4546): protocol: write tcp 188.166.225.170:43578->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:43578->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43578->145.79.227.23:4546: use of closed network connection Nov 11 05:12:35.730098044 (45h8m34.281568505s) [noise=true , address=145.79.227.24:4581 , piece_id=5PUE4EYDSSDJKICNL6BWTDIYESTHBGCA2D55BPPGVCIUAAXCFU7Q, seg_pos={0 0}, upload=39782]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:40602->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:40602->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40602->145.79.227.24:4581: use of closed network connection Nov 11 05:12:35.730230146 (45h8m34.281700607s) [noise=true , address=145.79.227.27:4689 , piece_id=AYSB5RC47IOX5O75VXTXMRFMMNKFTFXVACAX5E7PEGFOWUJEHRWA, seg_pos={0 0}, upload=39782]: Done uploading piece. err: ecclient: upload failed (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ, address:145.79.227.27:4689): protocol: write tcp 188.166.225.170:41488->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:41488->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41488->145.79.227.27:4689: use of closed network connection Nov 11 05:12:35.730360994 (45h8m34.281831455s) [noise=true , address=145.79.227.30:4761 , piece_id=VAKE56MU533FSWDIRDZO4FO4ULG4RIYIZX6PZK5QCSJSPER54S3Q, seg_pos={0 0}, upload=39782]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:52462->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:52462->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52462->145.79.227.30:4761: use of closed network connection Nov 11 05:12:35.730593046 (45h8m34.282063547s) [noise=true , address=145.79.227.31:4797 , piece_id=EY2RHNOF7GCHLERQQW7CCFEMVAZDEZ6KU3GYA4XVBBL26SBWQLPA, seg_pos={0 0}, upload=39782]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:145.79.227.31:4797): protocol: write tcp 188.166.225.170:57888->145.79.227.31:4797: use of closed network connection; write tcp 188.166.225.170:57888->145.79.227.31:4797: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57888->145.79.227.31:4797: use of closed network connection Nov 11 05:12:35.730752019 (45h8m34.28222248s) [noise=true , address=145.79.227.8:4331 , piece_id=A25M674WCTR4HXXH44AVWLU7NDJHJIHQYVZDVH6CKTP36ZLU3P4A, seg_pos={0 0}, upload=39782]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 188.166.225.170:33390->145.79.227.8:4331: use of closed network connection; write tcp 188.166.225.170:33390->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33390->145.79.227.8:4331: use of closed network connection Nov 11 05:12:35.730789952 (45h8m34.282260413s) [seg_pos={0 0}, upload=39782]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 11 05:12:35.982981004 (45h8m34.534451455s) [upload=39782]: Done uploading segments... Nov 11 05:12:35.983025178 (45h8m34.534495639s) [upload=39782]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.25554205s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.53498585s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:12:33 UTC) orbiter warning: context deadline exceeded Duration: 3.535014544s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 318.064719ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:12:26 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 318.093043ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.437759198s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:12:06 UTC) orbiter warning: context deadline exceeded Duration: 3.437792211s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 328.413735ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:11:34 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 328.43732ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 793.594222ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:11:12 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 793.65602ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.397115524s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:10:53 UTC) orbiter warning: context deadline exceeded Duration: 3.397276049s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.395871626s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:10:32 UTC) orbiter warning: context deadline exceeded Duration: 3.395906754s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 344.945314ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:10:19 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 344.973348ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.425856925s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:10:16 UTC) orbiter warning: context deadline exceeded Duration: 3.425917861s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.435114877s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:10:00 UTC) orbiter warning: context deadline exceeded Duration: 3.435149683s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 337.44354ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:09:22 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 337.483275ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.397226171s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:08:57 UTC) orbiter warning: context deadline exceeded Duration: 3.397255005s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 339.641684ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:08:30 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 339.680588ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [...........X...] duration: 321.23661ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:07:47 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 321.408537ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.431958374s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:07:44 UTC) orbiter warning: context deadline exceeded Duration: 3.431981597s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [............X..] duration: 315.777795ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:06:51 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 315.815998ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.530126256s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:06:29 UTC) orbiter warning: context deadline exceeded Duration: 3.530244993s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [.............X.] duration: 325.511881ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:05:57 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 325.541237ms core website
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect) antiblip: 1 out of 15 is failed (threshold=6) [..............X] duration: 224.461951ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/dcs-satellites trace: https://jaeger.dev.storj.io/trace/4a868fd039d8a2c0 FAIL (2025-11-11 05:05:41 UTC) orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:50748->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 224.489603ms core website
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.368008185s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:05:31 UTC) orbiter warning: context deadline exceeded Duration: 3.368036398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.454186491s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:04:38 UTC) orbiter warning: context deadline exceeded Duration: 3.454216468s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.457743729s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:03:58 UTC) orbiter warning: context deadline exceeded Duration: 3.457957345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.410017192s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:03:10 UTC) orbiter warning: context deadline exceeded Duration: 3.410062368s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.485987638s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:02:27 UTC) orbiter warning: context deadline exceeded Duration: 3.486087439s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.513177765s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:01:29 UTC) orbiter warning: context deadline exceeded Duration: 3.513239713s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.615597629s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 05:00:21 UTC) orbiter warning: context deadline exceeded Duration: 3.615633276s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.416364801s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 04:59:15 UTC) orbiter warning: context deadline exceeded Duration: 3.416394749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.393636615s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 04:58:02 UTC) orbiter warning: context deadline exceeded Duration: 3.393667393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.559064704s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 04:57:34 UTC) orbiter warning: context deadline exceeded Duration: 3.559091444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.368087283s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 04:56:34 UTC) orbiter warning: context deadline exceeded Duration: 3.368124163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.476815705s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 04:55:20 UTC) orbiter warning: context deadline exceeded Duration: 3.476865799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.40687814s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 04:54:20 UTC) orbiter warning: context deadline exceeded Duration: 3.406912905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.420402292s trace: https://jaeger.dev.storj.io/trace/689f287eb356f93 FAIL (2025-11-11 04:53:19 UTC) orbiter warning: context deadline exceeded Duration: 3.420436517s core uplink us-select-1 us1-placement-20

1
...
11985
11986
11987
11988
11989
...
16862