Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XX.XX....................................] duration: 5.782582987s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:40:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.782615589s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.912285467s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:39:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.912301317s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 79.394604ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4f5cc7baf01c69e2 FAIL (2025-11-11 05:39:43 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:35746->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 79.429119ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XX.XX...................................] duration: 6.059252264s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:39:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.059274616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.639514577s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:38:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.639545166s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 77.009461ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4f5cc7baf01c69e2 FAIL (2025-11-11 05:38:55 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:35746->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 77.045129ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XX.XX..................................] duration: 5.73091796s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:38:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.730948397s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.807869278s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:38:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.807902822s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.635707685s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:38:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.635739796s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 56.061291ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4f5cc7baf01c69e2 FAIL (2025-11-11 05:37:41 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:35746->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 56.092902ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XX.XX.................................] duration: 5.873636555s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:37:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.873667984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.646048639s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:37:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.646095748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................XX.XX................................] duration: 5.677126748s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:37:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.677156103s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.........X.....] duration: 77.580069ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4f5cc7baf01c69e2 FAIL (2025-11-11 05:36:55 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:35746->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 77.605896ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................XX.XX...............................] duration: 5.842692739s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:36:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.842721294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.845093049s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:36:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.84512524s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................XX.XX..............................] duration: 5.755826277s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:36:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.755855243s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [..........X....] duration: 58.645673ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4f5cc7baf01c69e2 FAIL (2025-11-11 05:35:49 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:35746->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 58.676432ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................XX.XX.............................] duration: 5.873205259s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:35:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.873247691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.581327176s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:35:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.58134967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................XX.XX............................] duration: 5.781825503s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:35:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.781859238s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [X..........X...] duration: 80.224173ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4f5cc7baf01c69e2 FAIL (2025-11-11 05:34:59 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:35746->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 80.256836ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................XX.XX...........................] duration: 5.838331342s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:34:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.838361691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.653992544s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:34:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.654011301s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [.X..........X..] duration: 85.225637ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4f5cc7baf01c69e2 FAIL (2025-11-11 05:34:26 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:35746->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 85.276094ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................XX.XX..........................] duration: 6.005294968s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:34:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.005393227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................XX.XX.........................] duration: 5.887994294s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:33:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.888026826s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.589501965s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:33:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.589530098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................XX.XX........................] duration: 5.753784552s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:33:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.753814198s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [..X..........X.] duration: 66.631509ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4f5cc7baf01c69e2 FAIL (2025-11-11 05:33:22 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:35746->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 66.683159ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.769976977s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:33:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.769993868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.650853608s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:32:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.650874868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................XX.XX.......................] duration: 5.729672204s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:32:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.729703615s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 2 out of 15 is failed (threshold=6) [...X..........X] duration: 423.891485ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/4f5cc7baf01c69e2 FAIL (2025-11-11 05:32:42 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:35746->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 424.179163ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XX.XX......................] duration: 5.9441512s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:32:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.944175415s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [....X..........] duration: 91.483815ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/2922a504fa9810e5 FAIL (2025-11-11 05:32:03 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57968->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 91.55486ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.774975732s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:32:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.775007663s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XX.XX.....................] duration: 5.821842569s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:31:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.821872834s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.705171259s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:31:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.705206136s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.....X.........] duration: 69.320651ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/2922a504fa9810e5 FAIL (2025-11-11 05:31:22 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57968->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 69.349686ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [......X........] duration: 82.209693ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/2922a504fa9810e5 FAIL (2025-11-11 05:31:04 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57968->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 82.440724ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.67988693s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:31:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.679975007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................XX.XX....................] duration: 5.977872145s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:30:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.977905618s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.765828973s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:30:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.765861505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XX.XX...................] duration: 5.983020992s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:30:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.983050839s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [.......X.......] duration: 63.920357ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/2922a504fa9810e5 FAIL (2025-11-11 05:29:51 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57968->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 63.946959ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.746227711s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:29:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.746258381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XX.XX..................] duration: 6.055799339s trace: https://jaeger.dev.storj.io/trace/6379438e4d72cba5 FAIL (2025-11-11 05:29:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nt. Inline: false Nov 11 05:15:36.773720216 (45h11m35.325190677s) [upload=39830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 05:15:36.773727991 (45h11m35.325198453s) [upload=39830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 05:15:37.025556873 (45h11m35.577027334s) [seg_pos={0 0}, upload=39830]: Begin upload segment... Nov 11 05:15:37.025620234 (45h11m35.577090695s) [seg_pos={0 0}, upload=39830]: Done begin upload segment. Nov 11 05:15:37.025658146 (45h11m35.577128607s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.026621430 (45h11m35.578091891s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027004370 (45h11m35.578474831s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027428136 (45h11m35.578898597s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.027828499 (45h11m35.57929896s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028198533 (45h11m35.579668994s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.028883297 (45h11m35.580353758s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029294270 (45h11m35.580764731s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029607797 (45h11m35.581078258s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.029949889 (45h11m35.58142034s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030296459 (45h11m35.58176692s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030626697 (45h11m35.582097158s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.030957097 (45h11m35.582427548s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031303235 (45h11m35.582773696s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.031631190 (45h11m35.583101651s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:37.032071940 (45h11m35.583542391s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Uploading piece... Nov 11 05:15:38.771377494 (45h11m37.322847955s) [upload=39830]: Next returned nil segment Nov 11 05:15:38.771395228 (45h11m37.322865689s) [upload=39830]: Waiting for error group managing segments... Nov 11 05:15:39.102527199 (45h11m37.65399765s) [noise=true , address=45.91.116.8:4045 , piece_id=WQ67S5AJFMMWN5SA7YOMCP226NNNSMLQJ45PD3LHHF2GOGWQATTA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.119177184 (45h11m37.670647655s) [noise=true , address=77.90.31.13:4627 , piece_id=QXV3USNVAU6UQETFHRAFULRSR7FMH2TRTJHO3WBQ2IU4VFRG3WNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.126230976 (45h11m37.677701437s) [noise=true , address=45.91.116.11:4118 , piece_id=7JND6YQ24NQ2BHLZYWZHFYLEY63YKANLV5OY2H3CC5YMI3LZMOYA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.141913097 (45h11m37.693383558s) [noise=true , address=45.91.116.13:4153 , piece_id=6INRGMRLHGK6VL2I6GYAAB5QXZ4SINCSD3XDXV4ECAOXBBCV7PCA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.175348025 (45h11m37.726818486s) [noise=true , address=45.91.116.4:4940 , piece_id=UCM67BZGLRKYXMSWFTLAT6B5BD2OCDJGT37RMAFZIPQ6GJB7C6RQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.179830029 (45h11m37.731300491s) [noise=true , address=77.90.31.18:4807 , piece_id=3KNJQQEHN6LXRYGZVMGZBBE4CXDXXBVHQTOSBKHRSFQU2LE2PBAQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.182173892 (45h11m37.733644343s) [noise=true , address=45.91.116.13:4155 , piece_id=BVV4SHYLWSZCGFRGEM4EPAIIDPW3TCESTNBK3WBDL2XDRU6IHUNQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.199649730 (45h11m37.751120191s) [noise=true , address=77.90.31.12:4590 , piece_id=ARWEVSKUC3H5B6DXPETDA24EGAF4WSFEQV357TMNHHSIIEKQ7TWA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:15:39.207214955 (45h11m37.758685416s) [noise=true , address=77.90.31.14:4661 , piece_id=CMBA2OIGJ65SV2PI5RUS4HOJW5XWOU2BUK2SHW4AMUXSKPRXVDMQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:05.601069625 (45h12m4.152540096s) [noise=true , address=145.79.227.9:4367 , piece_id=AMPAMTEEQU6APB4J67KIXSGTMKYOLBKAM2H7EPQILP4PAPU4WUEA, seg_pos={0 0}, upload=39830]: Done uploading piece. err: <nil> Nov 11 05:16:06.066368999 (45h12m4.61783948s) [noise=true , address=145.79.227.17:4972 , piece_id=3ARWDUSQKNIS7B4GPXICJ5XRCAVPQSWYHODAHZFPWIVONMVFJAEQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51768->145.79.227.17:4972: use of closed network connection Nov 11 05:16:06.066392924 (45h12m4.617863395s) [noise=true , address=145.79.227.18:4006 , piece_id=TJRTQEM5MDDFICFMMW6QW2GKAF6WWEANPK6ZSACOZSH2GOD4LH7A, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55838->145.79.227.18:4006: use of closed network connection Nov 11 05:16:06.066528302 (45h12m4.617998763s) [noise=true , address=145.79.227.2:4114 , piece_id=ADCL6CV36E3NHDXGD2YH5SMFSSJ5HDC43MO3DVT23PTJ34ONBNLQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45876->145.79.227.2:4114: use of closed network connection Nov 11 05:16:06.066687405 (45h12m4.618157856s) [noise=true , address=145.79.227.27:4690 , piece_id=T7PITNHESOVYQCVLGV2WODYM3I2DDZGRZEO4JJRNQBQR4IANOSKQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55048->145.79.227.27:4690: use of closed network connection Nov 11 05:16:06.066843573 (45h12m4.618314054s) [noise=true , address=145.79.227.10:4401 , piece_id=PU4CPX43A5BDRE56R3HWFEYUWIJRQLIZQUJUP4BIIUSMGIPK6KGQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, address:145.79.227.10:4401): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:16:06.066981726 (45h12m4.618452177s) [noise=true , address=145.79.227.26:4655 , piece_id=453STAWTU6YDKX4BECFFKUWUM6JOVKMBG4XD7KRWCBQVL3NCBCZQ, seg_pos={0 0}, upload=39830]: Done uploading piece. err: ecclient: upload failed (node:1xpZx5VbgHmbepaS9ZXgHfp7SZGeYPEjuJddQeYPUVpDU4EGx5, address:145.79.227.26:4655): protocol: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48168->145.79.227.26:4655: use of closed network connection Nov 11 05:16:06.067032913 (45h12m4.618503374s) [seg_pos={0 0}, upload=39830]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Nov 11 05:16:06.312833376 (45h12m4.864303837s) [upload=39830]: Done uploading segments... Nov 11 05:16:06.312885866 (45h12m4.864356327s) [upload=39830]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.055843232s core uplink us-select-1
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io) antiblip: 1 out of 15 is failed (threshold=6) [........X......] duration: 53.525564ms remote: "tcp:[2620:cb:2000::1]:443" target: https://www.storj.io/ trace: https://jaeger.dev.storj.io/trace/2922a504fa9810e5 FAIL (2025-11-11 05:28:57 UTC) orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:57968->[2620:cb:2000::1]:443: read: connection reset by peer Duration: 53.544369ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.650451485s trace: https://jaeger.dev.storj.io/trace/63ed88b300e27386 FAIL (2025-11-11 05:28:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 77]: Begin upload segment... Nov 11 05:11:40.051871659 (45h7m38.60334212s) [seg_pos={0 0}, upload=39777]: Done begin upload segment. Nov 11 05:11:40.051912387 (45h7m38.603382848s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.053641027 (45h7m38.605111479s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054037754 (45h7m38.605508215s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.054433046 (45h7m38.605903507s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055097110 (45h7m38.606567561s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055544062 (45h7m38.607014513s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.055898687 (45h7m38.607369148s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056205511 (45h7m38.607675972s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.056564404 (45h7m38.608034865s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057062313 (45h7m38.608532764s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057425815 (45h7m38.608896266s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.057782744 (45h7m38.609253205s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058329806 (45h7m38.609800257s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.058778230 (45h7m38.610248691s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059143857 (45h7m38.610614308s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059492851 (45h7m38.610963312s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.059893183 (45h7m38.611363634s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:40.060397354 (45h7m38.611867825s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Uploading piece... Nov 11 05:11:41.436493233 (45h7m39.987963694s) [upload=39777]: Next returned nil segment Nov 11 05:11:41.436511006 (45h7m39.987981467s) [upload=39777]: Waiting for error group managing segments... Nov 11 05:11:41.785470616 (45h7m40.336941077s) [noise=true , address=5.231.73.3:4025 , piece_id=GLCYHJVYU3K4TLQRV5GKXCTNWYD6U56LDDF3TSBRN7FGV5Q5KDCQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.797122653 (45h7m40.348593124s) [noise=true , address=5.231.73.4:4027 , piece_id=UH4C2OD4GTISXJVJ4NLCJH5V3QKSVSKIOS4CAYULMZ77OGKCIORQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.834411449 (45h7m40.38588191s) [noise=true , address=64.49.13.137:4497 , piece_id=EH6EP6GB2FTNPNA6SRE2ISL7YRG4YZMT2Z23LC6Y7DX66C5H5COQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.838130059 (45h7m40.38960052s) [noise=true , address=5.231.73.2:4033 , piece_id=R67BMRRYTXH3DW3X4XAELWCRJF5WX5YYGEWEP6VVTJSEQK4CENKA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:41.977355533 (45h7m40.528825994s) [noise=true , address=5.231.73.5:4036 , piece_id=HSXPEVXXQGQVURSW2FCW5OPTMAI55EOCGEGD2PGQBM2N3MBU7JWQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.024819354 (45h7m40.576289825s) [noise=true , address=77.90.31.15:4708 , piece_id=AYUNE6RLCKNFRMJHYY7U3GUZSFBLXSCAU72BSMHS67NT7TXJ4PEA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.050212707 (45h7m40.601683169s) [noise=true , address=77.90.31.16:4746 , piece_id=Y7H77MEJOVBM4HFNJOJVB5JEXTVS3WPDG26QK3OSKEGTHKPO6ICA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.086186298 (45h7m40.637656749s) [noise=true , address=45.91.116.14:4209 , piece_id=SJOWEWOMMETVXBQ4HW46WL2FJLNUZUT3NTEZCLXWFHV4GHMX3SBQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.087683699 (45h7m40.63915415s) [noise=true , address=77.90.31.11:4563 , piece_id=MARHY3N6HWFHAH7QOFZ5FFQ4GETM4PI7TL3EO4KLBNRGGSFVW2JQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.088758226 (45h7m40.640228687s) [noise=true , address=45.91.116.6:4995 , piece_id=JPZV5XZOP2BKYNH3PPONAL3QTXZY6D3GZ4KAIFBB4LHPNIZASXJQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.097480505 (45h7m40.648950966s) [noise=true , address=77.90.31.10:4522 , piece_id=FKL7Q3D6JIQWRK3RB544TQJLYCXW5DH2QMWXURDNE5G2B5PBK35A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:11:42.431022084 (45h7m40.982492545s) [noise=true , address=45.91.116.10:4110 , piece_id=DV6XIOEFSJFYM6MJJGLVJAHDYEX7Y2U5FXMFCXILEYHYKTMTR6MA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:36.046503955 (45h8m34.597974416s) [noise=true , address=145.79.227.7:4298 , piece_id=2IH55SSE7QQCEKZPJ3RYKJORPUFDQIEQMTLZZCJNQIEDNBW5H7NA, seg_pos={0 0}, upload=39777]: Done uploading piece. err: <nil> Nov 11 05:12:38.976413987 (45h8m37.527884458s) [noise=true , address=145.79.227.38:4993 , piece_id=HFMYULB5MNXWCVAKCCFY66VL3WEPGHWS4DQNDQUA6VJLMZBTMB7A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12UCrrdWBnenaoUHe19vCMZaWDiZ39hRxjjp6qWe9NmmGMjRK1c, address:145.79.227.38:4993): protocol: expected piece hash; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; write tcp 188.166.225.170:59772->145.79.227.38:4993: use of closed network connection; EOF Nov 11 05:12:38.976439646 (45h8m37.527910107s) [noise=true , address=145.79.227.16:4915 , piece_id=64RYR3QB7M2ND73AFADRYYTXLBJNWBPHACRFMWGXHDDQMAT2N6ZQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:12fqZipCJaNd9u5TvdWZ9nJFs26MiGtYX3Bbh41eQWP1Z4eJRBt, address:145.79.227.16:4915): protocol: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.16:4915: use of closed network connection Nov 11 05:12:38.976586455 (45h8m37.528056926s) [noise=true , address=145.79.227.41:4117 , piece_id=4FGT6DLHX4NXHILEQCYYNMYJ4YO4L4GVQGDGQVIQA3PQXBFYCHEQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:1QVEhFpjfLubGsZmoY1XQdDWevLy2xWdjtPq4JnbSYuBgEXgz7, address:145.79.227.41:4117): protocol: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38774->145.79.227.41:4117: use of closed network connection Nov 11 05:12:38.976836331 (45h8m37.528306792s) [noise=true , address=145.79.227.39:4039 , piece_id=SQ52S37ROLBWCXHTEE54JIYBR6K5GX4IW3IIS436X4JWVOADUGTQ, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:16oTDRhBhfgL5tFgu7Y36Yammf1baype6QsgtTf5zCtdpxEcbn, address:145.79.227.39:4039): protocol: expected piece hash; context deadline exceeded; EOF Nov 11 05:12:38.976964916 (45h8m37.528435387s) [noise=true , address=145.79.227.31:4832 , piece_id=UZBC6HLQWPOIGV4T3UZJJKEQTGOZYRFMCXDQS6S46XBRBISPZW4A, seg_pos={0 0}, upload=39777]: Done uploading piece. err: ecclient: upload failed (node:124qAWHeoHzitvbbGrzwBMn6G8xKQfwBQBjrQTducaeVWLiEpcP, address:145.79.227.31:4832): protocol: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44798->145.79.227.31:4832: use of closed network connection Nov 11 05:12:38.977001626 (45h8m37.528472087s) [seg_pos={0 0}, upload=39777]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Nov 11 05:12:39.244611519 (45h8m37.79608198s) [upload=39777]: Done uploading segments... Nov 11 05:12:39.244711339 (45h8m37.7961818s) [upload=39777]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.650482975s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)